2024-Parameter Choice and Neuro-Symbolic Approaches for Deep Domain-Invarian

As artificial intelligence (AI) systems advance, we are moving towards broad AI capable of performing well on diverse tasks, understanding context, and adapting rapidly to new scenarios. Broad AI, as conceptualized by Hochreiter (2022), represents a significant advancement over current narrow AI systems. It aims to create sophisticated and adaptive systems characterized by enhanced capabilities in knowledge transfer and interaction, adaptability and robustness, abstraction and advanced reasoning, and efficiency. These systems are designed to quickly adapt to new situations, customers, products, processes, workflows, or sensory inputs, addressing the limitations of current deep learning approaches which often underperform in real-world applications due to their data-hungry nature and limited transfer capabilities. Key challenges in developing such broad AI systems include distribution shifts and generalization. Addressing these challenges is crucial for creating AI systems that can solve a wide range of tasks across various domains and overcome the limitations that currently hinder the widespread adoption of AI in critical real-world scenarios. Neuro-symbolic (NeSy) AI emerges as a promising approach to tackle these challenges by bridging the gap between symbolic and sub-symbolic paradigms of AI (Marcus, 2001, 2020). It combines the strengths of both approaches while mitigating their individual weaknesses. Symbolic AI excels in logical reasoning, interpretability, and explicit knowledge representation, but struggles with adaptability and learning from raw data. Sub-symbolic AI, exemplified by deep neural networks, excels in pattern recognition and learning from large datasets, but often lacks interpretability and struggles with abstract reasoning. NeSy AI aims to create synergy between these approaches by integrating neural learning with symbolic knowledge and reasoning (Garcez et al., 2008, 2015; Besold et al., 2017; Garcez et al., 2019; Garcez and Lamb, 2020; Lamb et al., 2020; Hamilton et al., 2022; Yu et al., 2023). This integration is challenging due to the fundamentally different nature of these paradigms, which has historically limited their combination. However, recent advances in machine learning - specifically due to advances made by large language models (LLMs) (Vaswani et al., 2017; Wei et al., 2022; Singhal et al., 2023; Schick et al., 2023) - and knowledge representation (Augusto, 2022) have made it possible to create hybrid systems that leverage the strengths of both approaches. LLMs provide a comprehensive natural language understanding and generalization capability, and enable a seamless combination of logical reasoning with statistical approaches. NeSy AI systems can potentially offer improved adaptability, robustness, generalizability, and interpretability compared to purely symbolic or sub-symbolic systems. These characteristics are essential for developing broad AI systems, which are designed to solve a wide range of tasks across various domains and adapt to new challenges without extensive re-training or fine-tuning. Therefore, to create broad AI systems, it is essential to advance the field of DA and, in particular, domain-invariant learning.

随着人工智能（AI）系统的进步，我们正朝着能够在多样化任务中表现良好、理解上下文并快速适应新场景的广泛AI迈进。正如Hochreiter（2022）所构想的，广泛AI代表了对当前狭窄AI系统的重大进步。它旨在创建复杂且自适应的系统，这些系统在知识转移和交互、适应性和鲁棒性、抽象和高级推理以及效率方面具有增强的能力。这些系统旨在快速适应新情况、新客户、新产品、新流程、新工作流或感官输入，以解决当前深度学习方法的局限性，因为这些方法通常由于其对数据的高度依赖和有限的转移能力而在实际应用中表现不佳。开发此类广泛AI系统的关键挑战包括分布转移和泛化。解决这些挑战对于创建能够解决各种领域广泛任务的AI系统至关重要，并克服目前阻碍AI在关键现实场景中广泛应用的限制。神经符号（NeSy）AI作为一种有前景的方法，旨在通过弥合符号和子符号AI范式之间的差距来应对这些挑战（Marcus, 2001, 2020）。它结合了两种方法的优势，同时减轻了各自的弱点。符号AI在逻辑推理、可解释性和显式知识表示方面表现出色，但在适应性和从原始数据中学习方面存在困难。子符号AI以深度神经网络为例，在模式识别和从大数据集中学习方面表现优异，但通常缺乏可解释性，并且在抽象推理方面存在困难。NeSy AI旨在通过将神经学习与符号知识和推理相结合，创造这些方法之间的协同效应（Garcez et al., 2008, 2015; Besold et al., 2017; Garcez et al., 2019; Garcez and Lamb, 2020; Lamb et al., 2020; Hamilton et al., 2022; Yu et al., 2023）。由于这些范式的本质差异，这种整合具有挑战性，这在历史上限制了它们的结合。然而，最近在机器学习方面的进展——尤其是大型语言模型（LLMs）所取得的进展（Vaswani et al., 2017; Wei et al., 2022; Singhal et al., 2023; Schick et al., 2023）——以及知识表示（Augusto, 2022）的进展，使得创建利用两种方法优势的混合系统成为可能。LLMs提供了全面的自然语言理解和泛化能力，并能够无缝地将逻辑推理与统计方法结合起来。与纯符号或子符号系统相比，NeSy AI系统有可能提供更好的适应性、鲁棒性、泛化能力和可解释性。这些特征对于开发广泛AI系统至关重要，这些系统旨在解决各种领域的广泛任务，并能够在没有广泛再训练或微调的情况下适应新挑战。因此，为了创建广泛AI系统，推动DA领域的发展，特别是领域不变学习，是至关重要的。

In contrast to traditional single-model approaches, NeSy AI systems leverage multiple models and methods to address domain-specific requirements and generalize across varying conditions. They can select between parametric, non-parametric and symbolic methods that incorporate symbolic reasoning up to powerful pattern recognition abilities, structured knowledge bases, and search engines. This diverse toolkit enables more adaptable and interpretable methodologies. A hybrid paradigm also enables the formulation of higher-level abstractions and logical expressions through dedicated symbolic approaches. It incorporates learning-for-reasoning methods to treat the learning aspect as an accelerator for reasoning, in which deep neural networks are employed to reduce the search space for symbolic systems (Silver et al., 2016, 2017b,a; Qu and Tang, 2019; Schrittwieser et al., 2020). Reasoning-for-learning that views reasoning as a way to regularize learning, in which symbolic knowledge acts as a guiding constraint that oversees machine learning tasks (Hu et al., 2016; Xu et al., 2018), and learning-reasoning to enable a symbiotic relationship between learning and reasoning to boost problem-solving capabilities (Donadello et al., 2017; Manhaeve et al., 2018; Mao et al., 2019; Ellis, 2023). While NeSy AI systems offer numerous advantages, their implementation presents significant challenges. In practice, building such hybrid systems is challenging due to the inherent complexity of integrating disparate symbolic and neural components, as well as the difficulty in designing interfaces that allow seamless interaction between these components. This complexity often results in ad-hoc implementations that scale poorly and don't generalize across a variety of tasks and domains. The lack of standardized frameworks for NeSy integration further exacerbates these issues, leading to solutions that are often specific to particular problem domains and difficult to adapt to new scenarios.

与传统的单模型方法相比，NeSy AI 系统利用多个模型和方法来满足特定领域的需求，并在不同条件下进行泛化。它们可以在参数化、非参数化和符号方法之间进行选择，这些方法结合了符号推理、强大的模式识别能力、结构化知识库和搜索引擎。这种多样化的工具包使得方法论更加灵活和可解释。混合范式还通过专门的符号方法使得更高层次的抽象和逻辑表达的形成成为可能。它结合了学习-推理方法，将学习视为推理的加速器，其中深度神经网络被用来减少符号系统的搜索空间（Silver et al., 2016, 2017b,a; Qu and Tang, 2019; Schrittwieser et al., 2020）。推理-学习将推理视为一种规范化学习的方式，其中符号知识作为指导约束，监督机器学习任务（Hu et al., 2016; Xu et al., 2018），以及学习-推理以促进学习和推理之间的共生关系，从而提升问题解决能力（Donadello et al., 2017; Manhaeve et al., 2018; Mao et al., 2019; Ellis, 2023）。尽管 NeSy AI 系统提供了许多优势，但其实施面临重大挑战。在实践中，构建这样的混合系统具有挑战性，因为将不同的符号和神经组件集成在一起的固有复杂性，以及设计允许这些组件之间无缝交互的接口的困难。这种复杂性往往导致临时实现，扩展性差，且无法在各种任务和领域中进行泛化。缺乏 NeSy 集成的标准化框架进一步加剧了这些问题，导致的解决方案通常特定于某些问题领域，且难以适应新场景。

Motivated by these limitations and the potential of NeSy AI, my work analyzes common DA and NeSy approaches, with a particular focus on deep domain-invariant learning. This analysis is conducted from two complementary perspectives: First, we address DA issues faced when dealing with restricted sample sizes and where gradient-based optimization is feasible, and propose novel model-selection and parameter choice methods (Zellinger et al., 2021; Dinu et al., 2023). Second, we establish a connection between in-context learning (Vaswani et al., 2017) - the ability of LLMs to adapt to new tasks through prompting - and domain-invariant learning. This approach focuses on NeSy methodologies in situations where tuning models is infeasible, introducing domain-specific adaptations without gradient-based updates (Dinu, 2022; Dinu et al., 2024a,b).

受到这些限制和NeSy AI潜力的启发，我的工作分析了常见的领域适应（DA）和神经符号（NeSy）方法，特别关注深度领域不变学习。该分析从两个互补的角度进行：首先，我们解决了在处理受限样本大小时面临的DA问题，以及在梯度优化可行的情况下，提出了新颖的模型选择和参数选择方法（Zellinger等，2021；Dinu等，2023）。其次，我们建立了上下文学习（Vaswani等，2017）——大型语言模型（LLMs）通过提示适应新任务的能力——与领域不变学习之间的联系。这种方法专注于在无法微调模型的情况下的NeSy方法，引入了无需基于梯度更新的领域特定适应（Dinu，2022；Dinu等，2024a,b）。

Implementing effective UDA methods requires careful selection of model architectures and hyperparameters. This section examines approaches to model selection and parameter choice that aim to optimize UDA performance across diverse domains, focusing on how these methods can be applied to various UDA techniques such as moment matching (Zellinger et al., 2017; Gretton et al., 2012), divergence-based (Li et al., 2020), or discriminative approaches (Ganin et al., 2016), among others.

实施有效的无监督域适应（UDA）方法需要仔细选择模型架构和超参数。本节考察了旨在优化不同域之间 UDA 性能的模型选择和参数选择方法，重点讨论这些方法如何应用于各种 UDA 技术，如时刻匹配（Zellinger et al., 2017；Gretton et al., 2012）、基于发散度的方法（Li et al., 2020）或区分性方法（Ganin et al., 2016）等。

Parameter Choice Problem Typical approaches to parameter choice in UDA first involve training a sequence of

m

models with different hyper-parameter settings. This often includes modifying parameters such as the learning rate or weighting factors of loss terms. A current best practice for a second step is then to apply a model selection method to obtain the best-performing model with the minimum risk on the target domain. However, the state-of-the-art model selection methods (Sugiyama et al., 2007; You et al., 2019; Zellinger et al., 2021) can only perform as well as the best model within the sequence and disregard the

m - 1

trained models,in spite of the computational power and time invested for training. Furthermore, the choice of models and parameters often hinges on the task-specific characteristics of the dataset, necessitating dedicated experimentation.

参数选择问题在统一领域适应（UDA）中，参数选择的典型方法首先涉及训练一系列具有不同超参数设置的

m

模型。这通常包括修改学习率或损失项的权重因子等参数。当前的最佳实践是在第二步应用模型选择方法，以获得在目标领域上具有最低风险的最佳表现模型。然而，最先进的模型选择方法（Sugiyama et al., 2007; You et al., 2019; Zellinger et al., 2021）只能表现得与序列中的最佳模型一样好，而忽视了

m - 1

训练的模型，尽管在训练中投入了计算能力和时间。此外，模型和参数的选择往往依赖于数据集的任务特定特征，因此需要专门的实验。

More precisely,we calculate an ensemble

f^{agg} = \sum_{k = 1}^{m} c_{k} f_{k}

with minimal target risk

R_{q} (f^{agg}) = min_{c_{1}, \dots c_{m} \in R} R_{q} (\sum_{k = 1}^{m} c_{k} f_{k})

. Here

f_{k}

denotes the models within the sequence and

c_{k}

the aggregation coefficients. The optimal aggregation risk is less than or equal to the risk of the best model obtained by model selection,i.e.,

R_{q} (f^{agg}) \leq R_{q} (f^{sel})

,because this model is included within the sequence of aggregated models. Using vector-valued least squares, the vector of optimal coefficients can be calculated as follows (see Section 2.2):

更准确地说，我们计算具有最小目标风险的集成

f^{agg} = \sum_{k = 1}^{m} c_{k} f_{k}

。这里

f_{k}

表示序列中的模型，

c_{k}

表示聚合系数。最优聚合风险小于或等于通过模型选择获得的最佳模型的风险，即

R_{q} (f^{agg}) \leq R_{q} (f^{sel})

，因为该模型包含在聚合模型的序列中。使用向量值最小二乘法，可以如下计算最优系数的向量（见第2.2节）：

where

f_{q} (x) = \int_{Y} y d q (y ∣ x)

is the Bayesian predictor (the model with minimal risk on the target domain). The Gram matrix

G

consists of scalar products between the predictions of the individual models within the sequence,i.e. that

G = {(\int_{X} {⟨ f_{k} (x), f_{u} (x) ⟩}_{Y} d q_{X} (x))}_{k, u = 1}^{m}

,and

g =

{(\int_{X} {⟨ f_{q} (x), f_{k} (x) ⟩}_{Y} d q_{X} (x))}_{k = 1}^{m}

. Since calculating

f_{q}

requires access to the labels of the target domain, to overcome this problem, under the assumptions of 1) a covariate shift and 2) a bounded density ratio between the marginals of the source distribution

p

and target distribution

q

,so that

β (x) = \frac{d q_{X}}{d p_{X}} (x) \in [0, B]

with some constant

B > 0

,we arrive at the following result:

其中

f_{q} (x) = \int_{Y} y d q (y ∣ x)

是贝叶斯预测器（在目标领域上具有最小风险的模型）。Gram 矩阵

G

由序列中各个模型预测之间的标量积组成，即

G = {(\int_{X} {⟨ f_{k} (x), f_{u} (x) ⟩}_{Y} d q_{X} (x))}_{k, u = 1}^{m}

和

g =

{(\int_{X} {⟨ f_{q} (x), f_{k} (x) ⟩}_{Y} d q_{X} (x))}_{k = 1}^{m}

。由于计算

f_{q}

需要访问目标领域的标签，为了解决这个问题，在以下假设下：1）协变量偏移和 2）源分布

p

和目标分布

q

之间的有界密度比，使得

β (x) = \frac{d q_{X}}{d p_{X}} (x) \in [0, B]

具有某个常数

B > 0

，我们得出以下结果：

We view in-context learning as deeply rooted in DA and related to domain-invariant learning, since LLMs seem to learn shared feature representations

φ_{trans}^{l} (\cdot)

that generalize across diverse tasks and domains without explicit retraining. This is because LLMs are trained on a large corpus of data that includes various prompt-to-prediction pairs from different domains. This training enables an LLMs to form domain-invariant representations that generalize across different tasks, even to those they have not specifically been trained for. In our paper Dinu et al. (2024a), we accredit this to the associations formed between different domains during training based on the instructions and contextual sequences for problem-specific tasks.

我们将上下文学习视为深深植根于数据增强（DA）并与领域不变学习相关，因为大型语言模型（LLMs）似乎学习了共享特征表示

φ_{trans}^{l} (\cdot)

，这些表示能够在不同任务和领域之间进行泛化，而无需明确的再训练。这是因为 LLMs 在一个包含来自不同领域的各种提示到预测对的大型数据集上进行训练。这种训练使得 LLMs 能够形成领域不变的表示，这些表示能够在不同任务之间进行泛化，即使是那些它们没有专门训练过的任务。在我们的论文 Dinu 等（2024a）中，我们将此归因于在训练过程中基于问题特定任务的指令和上下文序列形成的不同领域之间的关联。

We draw this conclusion based on the evidence of analyzing the memorization and learning capabilities of LLMs (Carlini et al., 2021; Min et al., 2022; Hartmann et al., 2023; Oswald et al., 2023; Duan et al., 2024), polysemanticity and capacity in neural networks (Scherlis et al., 2022; Zhao et al., 2024), and semantic feature spaces (Wolfram, 2023). To illustrate, Figure 1.2 presents a UMAP projection (McInnes et al., 2018) of the latent space, visualizing the semantic relationships between four different domains based on their embedded representations in the LLM. We show the embedding activations of the last layer of the open-source GPT-Neo 1.3 billion parameter model (Black et al., 2021) for the respective domains. Here, domain represents a specific data distribution of token sequences. These domains are denoted as Mathematical, Programming, Natural Language, and Random domain. Each dataset includes examples representative of the data distribution of token sequences for the respective domains.

我们基于对 LLM（Carlini et al., 2021; Min et al., 2022; Hartmann et al., 2023; Oswald et al., 2023; Duan et al., 2024）记忆和学习能力的分析证据、神经网络中的多义性和容量（Scherlis et al., 2022; Zhao et al., 2024）以及语义特征空间（Wolfram, 2023）得出这一结论。为此，图 1.2 展示了潜在空间的 UMAP 投影（McInnes et al., 2018），可视化了基于 LLM 中嵌入表示的四个不同领域之间的语义关系。我们展示了开源 GPT-Neo 13 亿参数模型（Black et al., 2021）最后一层的嵌入激活，针对各自领域。这里，领域代表特定的数据分布的标记序列。这些领域被称为数学、编程、自然语言和随机领域。每个数据集包括代表各自领域标记序列数据分布的示例。

However, we also see an overlap between the first three domains, since they have semantically related features,such as "

a = b

" and "

a

equals

b

". This is possible because during training, the model developed a semantic mapping that captures cross-domain correspondences and contextual synonyms. In-context learning prompts then interpolate between these feature representations by establishing new in-context associations between sequences and, consequently, between data distributions, thereby enforcing domain-invariant predictions. The power of in-context learning lies in its ability to generalize representations across various tasks and domains by leveraging semantically related features for context-specific adaptations of data distributions.

然而，我们也看到前三个领域之间存在重叠，因为它们具有语义相关的特征，例如“

a = b

”和“

a

等于

b

”。这是可能的，因为在训练过程中，模型发展了一种语义映射，捕捉跨领域的对应关系和上下文同义词。上下文学习提示通过在序列之间建立新的上下文关联，从而在这些特征表示之间进行插值，因此在数据分布之间强制执行领域不变的预测。上下文学习的力量在于其能够通过利用语义相关特征在各种任务和领域之间推广表示，以实现数据分布的上下文特定适应。

Structure of In-Context Learning for Domain Adaptation LLMs are usually trained on associated pairs with instructions

i \in I

,labeled contexts

c \in C

sampled w.r.t. to their marginal source and target distribution respectively,and unlabeled query statements

s \in S

. The contexts include expected labeled outcomes

y \in Y

that create a bridge between domains, and help the model understand relationships between sequences of tokens. Specifically, we denote the following elements:

上下文学习的结构用于领域适应，LLMs 通常在带有指令的关联对上进行训练

i \in I

，标记的上下文

c \in C

根据它们的边际源和目标分布分别进行采样，以及未标记的查询语句

s \in S

。这些上下文包括期望的标记结果

y \in Y

，它们在领域之间建立桥梁，并帮助模型理解标记序列之间的关系。具体而言，我们表示以下元素：

where a scaling factor

\sqrt{d_{k}}

is introduced to mitigate the effect of large dot products in high-dimensional spaces. This scaling helps maintain stable gradients during training,particularly for larger values of

d_{k}

,and prevents the softmax function from producing small gradients in regions where the input is large in magnitude.

其中引入了一个缩放因子

\sqrt{d_{k}}

，以减轻高维空间中大点积的影响。这个缩放有助于在训练过程中保持稳定的梯度，特别是对于较大的

d_{k}

值，并防止 softmax 函数在输入幅度较大时产生小梯度。

The self-attention function computes a weighted sum of the rows of the value matrix

V

(each row being a value vector),with weights derived from the similarity between corresponding rows of the query matrix

Q

and columns of the key matrix

K^{⊤}

. Here,self-attention refers to the fact that

Q, K, V

all share the same sequence matrix

z

. It’s worth noting that in other attention settings, different sequence representations can be used for each component. The resulting attention output

φ_{trans} (\cdot)

associates information from different parts of the input sequence, capturing the dependencies and relationships between tokens. For networks with multiple layered transformations

φ_{trans}^{l > 1}

the output is usually normalized before passed on to subsequent layers. The respective matrices

W_{Q}^{l}, W_{K}^{l}, W_{V}^{l}

learn these associations for domain-specific tasks and generalize to the semantics based on the instructions and contextual patterns. This generalization mechanism then later allows the model to extrapolate semantics across diverse domains when provided with few-shot examples, effectively enabling DA through the formation of context-dependent associations in the attention layers.

自注意力函数计算值矩阵

V

行的加权和（每行是一个值向量），权重来源于查询矩阵

Q

相应行与键矩阵

K^{⊤}

列之间的相似性。在这里，自注意力指的是

Q, K, V

所有共享相同的序列矩阵

z

。值得注意的是，在其他注意力设置中，可以为每个组件使用不同的序列表示。生成的注意力输出

φ_{trans} (\cdot)

关联了输入序列不同部分的信息，捕捉了标记之间的依赖关系和关系。对于具有多层变换的网络

φ_{trans}^{l > 1}

，输出通常在传递到后续层之前进行归一化。各自的矩阵

W_{Q}^{l}, W_{K}^{l}, W_{V}^{l}

学习这些关联以应对特定领域的任务，并根据指令和上下文模式进行语义泛化。这个泛化机制随后使模型能够在提供少量示例时跨不同领域推断语义，有效地通过在注意力层中形成上下文相关的关联来实现领域适应（DA）。

Domain-Invariance in Language Models In the previous sections, we have seen LLMs able of forming domain-invariant predictions based on our empirical observations, and accredited this to associative patterns learned during training. We now conclude our analysis by defining domain-invariance in an LLM setting, and propose a measurable relationship between semantic similarity of domains and the convergence of their embedded representations. This relationship forms the basis for future work on quantifying the models' capabilities for cross-domain generalization.

语言模型中的领域不变性在前面的部分中，我们已经看到大型语言模型（LLMs）能够基于我们的实证观察形成领域不变的预测，并将其归因于训练过程中学习的关联模式。我们现在通过在LLM环境中定义领域不变性来结束我们的分析，并提出领域语义相似性与其嵌入表示收敛之间的可测关系。这一关系为未来量化模型跨领域泛化能力的研究奠定了基础。

We note that in a general case an LLM is a function that maps

f_{θ} : P \to V^{*}

with parameters

θ \in Θ

,which produces a sequence of outputs based on the prompt. This is due to the auto-regressive setting of predicting and sampling the next token and appending it to the input, which is repeated until the halting criteria such as a stop token terminates the sequence. In this setting, a decoder typically samples according to some sampling strategy (i.e. using a temperature parameter, beam search, etc.) to obtain the most likely sequence of token. However, for simplicity the following definition will focus only on a special case where we omit the auto-regressive process and assume that the context

C \subset X \times V

considers the output space equal to the vocabulary,and the LLM predicts only the next token

V

. Furthermore,to define the domain invariance as shown in Example 1, we assume that "True" and "False" are tokens and part of the vocabulary and then formalize the definition as follows:

我们注意到，在一般情况下，LLM 是一个将

f_{θ} : P \to V^{*}

映射到具有参数

θ \in Θ

的函数，它根据提示生成一系列输出。这是由于自回归设置的原因，预测和采样下一个标记，并将其附加到输入中，直到停止标记等停止标准终止序列。在这种设置中，解码器通常根据某种采样策略（即使用温度参数、束搜索等）进行采样，以获得最可能的标记序列。然而，为了简化，以下定义将仅关注一个特殊情况，即我们省略自回归过程，并假设上下文

C \subset X \times V

将输出空间视为等于词汇表，并且 LLM 仅预测下一个标记

V

。此外，为了定义如示例 1 所示的领域不变性，我们假设“真”和“假”是标记并且是词汇表的一部分，然后将定义形式化如下：

Let

V

be the vocabulary of tokens of an LLM and

P \subset X \subset V^{*}

be the prompt space. Define

P = I^{m} \times C^{k_{p}} \times C^{k_{q}} \times S^{n}

with instruction space

i \in I \subset X

,context space

c \in C \subset X \times V

,and query space

s \in S \subset X

. Let

p (i, c, s)

and

q (i, c, s)

denote the joint distributions for the source and target domains respectively,with

p_{C}

and

q_{C}

being their respective marginal distributions over the context space. The sequence dimensions

m, k_{p}, k_{q}, n

are each

\geq 1

,the sequence length is

j = m + k_{p} + k_{q} + n

,and the first latent dimension is

d_{1} \geq 1

设

V

为 LLM 的标记词汇表，

P \subset X \subset V^{*}

为提示空间。定义

P = I^{m} \times C^{k_{p}} \times C^{k_{q}} \times S^{n}

具有指令空间

i \in I \subset X

、上下文空间

c \in C \subset X \times V

和查询空间

s \in S \subset X

。设

p (i, c, s)

和

q (i, c, s)

分别表示源域和目标域的联合分布，

p_{C}

和

q_{C}

为它们在上下文空间上的各自边际分布。序列维度

m, k_{p}, k_{q}, n

每个为

\geq 1

，序列长度为

j = m + k_{p} + k_{q} + n

，第一个潜在维度为

d_{1} \geq 1

。

Now,let an LLM be a function

f_{θ} : P \to V

with parameters

θ \in Θ

, with transformation layers

φ_{trans}^{l} : R^{j \times d_{l - 1}} \to R^{j \times d_{l}}

and dimensions

d_{l}

for

1 < l < L

,and the last transformation layer

φ_{trans}^{L} : R^{j \times d_{L - 1}} \to R^{d_{L}}

before the decoder

φ_{dec} : R^{d_{L}} \to V

,where the output dimension

d_{L} = | V | \geq 1

is equal to the vocabulary size.

现在，让一个 LLM 成为一个函数

f_{θ} : P \to V

，具有参数

θ \in Θ

，具有变换层

φ_{trans}^{l} : R^{j \times d_{l - 1}} \to R^{j \times d_{l}}

和维度

d_{l}

，用于

1 < l < L

，并且在解码器

φ_{dec} : R^{d_{L}} \to V

之前的最后一个变换层

φ_{trans}^{L} : R^{j \times d_{L - 1}} \to R^{d_{L}}

，其中输出维度

d_{L} = | V | \geq 1

等于词汇表大小。

Definition 1 (Domain-Invariant Language Models). We say an LLM has domain-invariant properties if, for all pairs of contextual samples with equivalent semantics,represented by

c_{p} \equiv c_{q}

,where

c_{p} \in C_{p}

comes from the marginal distribution

p_{C}

and the semantically equivalent

c_{q} \in C_{q}

comes from the marginal distribution

q_{C}

,the argmax (greedy sampling decoder) of their outputs from the last transformation layer

φ_{trans}^{L}

is equal. Formally:

定义 1（领域不变语言模型）。我们说一个 LLM 具有领域不变特性，如果对于所有具有等效语义的上下文样本对，表示为

c_{p} \equiv c_{q}

，其中

c_{p} \in C_{p}

来自边际分布

p_{C}

，而语义上等效的

c_{q} \in C_{q}

来自边际分布

q_{C}

，它们在最后一个变换层

φ_{trans}^{L}

的输出的 argmax（贪婪采样解码器）是相等的。形式上：

where

z_{p}^{L}

and

z_{q}^{L}

are the outputs of the last transformation layer derived from

c_{p}

and

c_{q}

respectively,with

z_{p}^{1} = φ_{enc} (0, c_{p}, 0, 0; ρ)

and

z_{q}^{1} = φ_{enc} (0, 0, c_{q}, 0; ρ)

Here,

φ_{enc} : P \to R^{j \times d_{1}}

is the embedding and concatenation function,

ρ : V \to R^{d_{1}}

is an embedding function,and0denotes the mask tokens for the components. The latent representations

Z_{p}^{l} = {z_{p, u}^{l} = φ^{1 : l} (c_{p, u})}_{u = 1}^{h_{p}}

and

Z_{q}^{l} = {z_{q, u}^{l} = φ^{1 : l} (c_{q, u})}_{u = 1}^{h_{q}}

come from the respective domains with

h_{p} \geq 1

and

h_{q} \geq 1

samples,by applying the composition function up to layer index

l

,where

φ^{1 : l} = φ_{trans}^{l} \circ φ_{trans}^{l - 1} \circ \dots \circ φ_{trans}^{2} \circ φ_{trans}^{1} \circ φ_{enc}

其中

z_{p}^{L}

和

z_{q}^{L}

是分别来自

c_{p}

和

c_{q}

的最后一个变换层的输出，具有

z_{p}^{1} = φ_{enc} (0, c_{p}, 0, 0; ρ)

和

z_{q}^{1} = φ_{enc} (0, 0, c_{q}, 0; ρ)

。这里，

φ_{enc} : P \to R^{j \times d_{1}}

是嵌入和连接函数，

ρ : V \to R^{d_{1}}

是嵌入函数，0 表示组件的掩码标记。潜在表示

Z_{p}^{l} = {z_{p, u}^{l} = φ^{1 : l} (c_{p, u})}_{u = 1}^{h_{p}}

和

Z_{q}^{l} = {z_{q, u}^{l} = φ^{1 : l} (c_{q, u})}_{u = 1}^{h_{q}}

来自各自的领域，具有

h_{p} \geq 1

和

h_{q} \geq 1

样本，通过应用组合函数直到层索引

l

，其中

φ^{1 : l} = φ_{trans}^{l} \circ φ_{trans}^{l - 1} \circ \dots \circ φ_{trans}^{2} \circ φ_{trans}^{1} \circ φ_{enc}

。

The semantic equivalence symbol

\equiv

refers to selecting consistently equivalent samples from the respective sets, e.g., the number " 1 " and string "one", which are then pairwise embedded, transformed, and compared. We then stipulate that for all these pairwise latent representations, their argmax must be equal, ensuring they map to the same next token. The determination of semantic equivalence is incumbent upon the modeling process of the distributions and is an integral part of the training process design. This definition illustrates how the phenomenon of cross-domain generalization in LLMs can emerge when a sufficiently large and diverse training data corpus is used. The definition of domain-invariant properties in LLMs and the semantic equivalence can be viewed as a language-based extension of the Curry-Howard correspondence, which originally establishes a direct relationship between computer programs and mathematical proofs (Howard, 1980).

语义等价符号

\equiv

指的是从各自集合中选择一致等价的样本，例如数字 "1" 和字符串 "one"，然后将它们成对嵌入、转换并比较。我们规定，对于所有这些成对的潜在表示，它们的 argmax 必须相等，以确保它们映射到相同的下一个标记。语义等价的确定依赖于分布的建模过程，是训练过程设计的一个 integral 部分。这个定义说明了在使用足够大且多样化的训练数据语料库时，LLMs 中跨领域泛化现象如何出现。LLMs 中领域不变属性的定义和语义等价可以被视为对 Curry-Howard 对应关系的基于语言的扩展，该对应关系最初建立了计算机程序与数学证明之间的直接关系（Howard, 1980）。

Domain-Invariant Associations While Definition (1) provides a strict criterion for domain invariance, in practice, we often have only a degree of similarity between domain representations, especially in an auto-regressive generative process. To quantify this similarity, we introduce a semantic distance measure that captures the closeness of representations of the model's layers,and provide an

ϵ

-close domain-invariance approximation.

领域不变关联虽然定义 (1) 为领域不变性提供了严格的标准，但在实践中，我们通常只有领域表示之间的相似度，特别是在自回归生成过程中。为了量化这种相似性，我们引入了一种语义距离度量，捕捉模型层表示的接近程度，并提供一个

ϵ

-接近的领域不变性近似。

We say that two domains with contextual samples

C_{p}

and

C_{q}

have

ϵ

- close domain-invariant associations if

d_{sem, θ} (C_{p}, C_{q}) \leq ϵ

,thereby setting an upper bound on the difference in representations measured by

d_{sem, θ}

. The semantic distance provides an approximation of the degree of domain invariance formed through associations, with smaller distances indicating greater similarity between domain representations. It also captures the idea that as the distance between the transformed representations of samples from two domains decreases, the semantic distance between these domains also decreases. The semantic distance

d_{sem, θ}

therefore quantifies this relationship and can be applied to a variety of domains, considering factors such as semantic overlap, structural similarity, and task-specific relevance.

我们说具有上下文样本

C_{p}

和

C_{q}

的两个领域具有

ϵ

- 接近的领域不变关联，如果

d_{sem, θ} (C_{p}, C_{q}) \leq ϵ

，从而为通过

d_{sem, θ}

测量的表示差异设定了上限。语义距离提供了通过关联形成的领域不变性的程度的近似，较小的距离表示领域表示之间的更大相似性。它还捕捉到这样一个观点：当来自两个领域的样本的变换表示之间的距离减小时，这两个领域之间的语义距离也会减小。因此，语义距离

d_{sem, θ}

定量化了这种关系，并可以应用于多种领域，考虑语义重叠、结构相似性和任务特定相关性等因素。

In practice, the argmax is not applied; instead, more sophisticated autoregressive sampling techniques are used. Therefore, it suffices that the last layer representations

z_{p}^{L}

and

z_{q}^{L}

are close enough to semantically map to a similar distribution of representations. This is supported by empirical evidence that suggests the existence of semantic meaning spaces or a semantic law of motion,as described by Stephen Wolfram (Wolfram,2023). The

ϵ

- close domain-invariant association offers a practical relaxation when perfect invariance is unattainable. This approach provides a quantifiable measure of similarity between domain representations. In an auto-regressive setting, it can leverage the semantic closeness between tokens to guide the semantic space towards the desired outcome.

在实践中，未应用 argmax；相反，使用了更复杂的自回归采样技术。因此，最后一层表示

z_{p}^{L}

和

z_{q}^{L}

只需足够接近，以在语义上映射到相似的表示分布。这得到了实证证据的支持，表明存在语义意义空间或语义运动法则，正如斯蒂芬·沃尔夫拉姆（Wolfram, 2023）所描述的那样。

ϵ

- 接近的领域不变关联在完美不变性无法实现时提供了一个实际的放宽。这种方法提供了领域表示之间相似性的可量化度量。在自回归设置中，它可以利用标记之间的语义接近性来引导语义空间朝向期望的结果。

This hypothesis formalizes the concept that when the semantic distance between samples of a source domain and a target domain is small, the model's discrepancy between its output distribution and the true output distribution on the target domain is bounded by its discrepancy on the source domain plus a small error term. This error term decreases as the semantic distance decreases.

该假设形式化了这样一个概念：当源域和目标域样本之间的语义距离较小时，模型的输出分布与目标域真实输出分布之间的差异受限于其在源域上的差异加上一个小的误差项。随着语义距离的减小，该误差项也会减小。

In turn, this also highlights the model's limitations: when a query extrapolates beyond the model's domain capabilities, we obtain meaningless and potentially hallucinated results (Huang et al., 2023, 2024), as the model attempts to fill knowledge gaps with its existing, but irrelevant, capabilities. To some degree, this limitation is often obfuscated by blending the generative process with reinforcement learning fine-tuning from human feedback (Ouyang et al., 2022; Kaufmann et al., 2024). However, recent publications have begun to critically examine these aspects (Hicks et al., 2024). Despite these and other open challenges, the proposed intersection of domain-invariant learning and LLMs, along with the utilization of their operator-like patterns, offers potential for grounding prompt engineering in a formal framework and advancing NeSy approaches, particularly in bridging symbolic reasoning with the adaptive capabilities of neural networks.

这也突显了模型的局限性：当查询超出模型的领域能力时，我们会得到无意义且可能是幻觉的结果（Huang et al., 2023, 2024），因为模型试图用其现有但无关的能力来填补知识空白。在某种程度上，这种局限性常常被通过将生成过程与来自人类反馈的强化学习微调相结合而掩盖（Ouyang et al., 2022; Kaufmann et al., 2024）。然而，最近的出版物开始批判性地审视这些方面（Hicks et al., 2024）。尽管存在这些和其他未解决的挑战，所提出的领域不变学习与大型语言模型（LLMs）的交集，以及利用其类操作符模式的潜力，为在正式框架中扎根提示工程并推进神经符号（NeSy）方法提供了可能性，特别是在将符号推理与神经网络的自适应能力相结合方面。

Building on the in-context learning capabilities of large language models (LLMs) and their ability to interpret in-context associations, we can define task-specific prompts that interpret various instructions and contextual patterns, and combine them in a neuro-symbolic (NeSy) approach. These prompts leverage the operator-like patterns learned by the model, such as equals, is like, and from ... to, to solve task-specific requirements without the necessity to update model parameters. This NeSy approach connects various domains via a language-based interface, utilizing the model's capacity to recognize and apply semantic patterns across domains.

基于大型语言模型（LLMs）的上下文学习能力及其解释上下文关联的能力，我们可以定义特定任务的提示，这些提示能够解释各种指令和上下文模式，并将其结合在一个神经符号（NeSy）方法中。这些提示利用模型学习到的类操作符模式，如等于、类似于和从...到，以解决特定任务的需求，而无需更新模型参数。这种NeSy方法通过基于语言的接口连接各个领域，利用模型识别和应用跨领域语义模式的能力。

We therefore consider LLMs suitable as semantic parsers (Dinu et al., 2024a) that can take in various expressions (which are operation descriptions defining how to manipulate strings), interpret them using learned associations, and predict the most probable statistical outcome. These expressions can be composed into complex functions, forming a multi-step generative process that leverages a model's domain-invariant capabilities.

因此，我们认为大型语言模型（LLMs）适合作为语义解析器（Dinu et al., 2024a），能够接受各种表达（这些是定义如何操作字符串的操作描述），利用学习到的关联进行解释，并预测最可能的统计结果。这些表达可以组合成复杂的函数，形成一个多步骤的生成过程，利用模型的领域不变能力。

The language-based interface is well-suited for a NeSy approach to interface with classical symbolic methods. We can interchange between solvers and various tools that can verify our predicted outcomes, taking advantage of the model's ability to bridge symbolic and sub-symbolic paradigms. This NeSy approach allows for more structured and rule-based approaches, where the model not only predicts statistical associations but also adheres to defined symbolic rules and expressions. This enhances our ability to verify results by combining the strengths of both neural and symbolic components.

基于语言的接口非常适合采用神经符号（NeSy）方法与经典符号方法进行交互。我们可以在求解器和各种工具之间进行互换，这些工具能够验证我们预测的结果，利用模型在符号和子符号范式之间架起桥梁的能力。这种神经符号方法允许采用更结构化和基于规则的方法，其中模型不仅预测统计关联，还遵循定义的符号规则和表达。这增强了我们通过结合神经和符号组件的优势来验证结果的能力。

SymbolicAI Framework In our work Dinu et al. (2024a), we propose a neuro-symbolic approach called SymbolicAI. The framework is designed to bridge the gap between symbolic reasoning and neural network capabilities, particularly with the help of LLMs.

符号人工智能框架在我们的工作中，Dinu et al. (2024a) 提出了一个称为符号人工智能（SymbolicAI）的神经符号方法。该框架旨在弥合符号推理与神经网络能力之间的差距，特别是在大型语言模型（LLMs）的帮助下。

At its core, SymbolicAI performs mappings that take in various symbols and produce symbols as output, where symbols are sequences of tokens drawn from a finite vocabulary

V

. These symbols can be of any length and structure, allowing for great flexibility in representation. The SymbolicAI framework uses these symbols to form more complex expressions, and uses in-context learning to adapt the generative process based on the given instructions, contexts and query statements. A key concept in this framework is the idea of expressions and their polymorphic contexts. An expression is an operation that defines transformations of symbols to create new symbols. This can be as simple as concatenating two symbols or as complex as applying mathematical operations to them. When this operation is evaluated, it produces a new symbol based on specific rules. For example, one rule might define how to perform arithmetic operations on symbols representing numbers and strings. To illustrate this, consider an example where we want to combine the symbols "eight" and "8". We could define a rule that interprets this as addition, resulting in a new symbol "16" as the prediction.

从本质上讲，SymbolicAI 执行映射，将各种符号作为输入并生成符号作为输出，其中符号是从有限词汇表中提取的标记序列

V

。这些符号可以具有任意长度和结构，从而在表示上提供了极大的灵活性。SymbolicAI 框架使用这些符号形成更复杂的表达式，并利用上下文学习根据给定的指令、上下文和查询语句来调整生成过程。该框架中的一个关键概念是表达式及其多态上下文的概念。表达式是定义符号转换以创建新符号的操作。这可以简单到连接两个符号，也可以复杂到对它们应用数学运算。当这个操作被评估时，它会根据特定规则生成一个新符号。例如，一个规则可能定义如何对表示数字和字符串的符号执行算术运算。为了说明这一点，考虑一个示例，我们想要组合符号“eight”和“8”。我们可以定义一个规则，将其解释为加法，结果生成一个新符号“16”作为预测。

The framework is grounded in formal language theory and grammars, building on the work of linguists and computer scientists Chomsky and Newell (Chomsky, 1959; Newell and Simon, 1972; Hays, 2003). Our NeSy approach uses symbolic methods to define sets of rules for relationships between objects, evoke causal structures, and perform formal evaluations. This allows us to define task-specific operations that query LLMs to guide their predictions and verify the results with other tools and solvers. An LLM can interpret instructions to form associative patterns with the respective data and return a statistically probable answer. Our framework then combines such operations into more complex expressions. These expressions guide the generative process of the LLM and enable efficient domain-specific adaptations even under continual domain changes without the necessity of parameter updates. SymbolicAI thus provides a systematic way to represent and manipulate symbolic information and control generative processes. Our NeSy approach is well suited for enabling enhanced reasoning capabilities and building broad AI systems that verify the results of generated content such as that obtained through LLMs, while still leveraging their powerful language understanding and generalization abilities. This can lead to more robust and versatile AI systems capable of handling a wider range of tasks, setting the foundation for automated AI workflows. Moreover, the symbolic component of NeSy systems can address challenges of computational irreducibility often encountered in complex tasks, allowing for more transparent, interpretable, and discernible results in scenarios where purely neural approaches might lead to unpredictable or difficult-to-verify outcomes.

该框架基于形式语言理论和语法，建立在语言学家和计算机科学家乔姆斯基和纽厄尔的研究基础上（乔姆斯基，1959；纽厄尔和西蒙，1972；海斯，2003）。我们的 NeSy 方法使用符号方法定义对象之间关系的规则集，唤起因果结构，并进行形式评估。这使我们能够定义特定任务的操作，以查询 LLM 来指导其预测，并用其他工具和求解器验证结果。LLM 可以解释指令，以形成与相应数据的关联模式并返回统计上可能的答案。然后，我们的框架将这些操作组合成更复杂的表达式。这些表达式指导 LLM 的生成过程，并使其能够在不断变化的领域中进行高效的领域特定适应，而无需更新参数。因此，SymbolicAI 提供了一种系统化的方式来表示和操控符号信息，并控制生成过程。我们的 NeSy 方法非常适合增强推理能力，并构建广泛的 AI 系统，以验证通过 LLM 生成的内容的结果，同时仍然利用它们强大的语言理解和概括能力。这可以导致更强大和多功能的 AI 系统，能够处理更广泛的任务，为自动化 AI 工作流奠定基础。此外，NeSy 系统的符号组件可以解决复杂任务中经常遇到的计算不可约性挑战，从而在纯神经方法可能导致不可预测或难以验证的结果的场景中，提供更透明、可解释和可辨别的结果。

We address the unsolved algorithm design problem of choosing a justified regularization parameter in unsupervised domain adaptation. This problem is intriguing as no labels are available in the target domain. Our approach starts with the observation that the widely-used method of minimizing the source error, penalized by a distance measure between source and target feature representations, shares characteristics with regularized ill-posed inverse problems. Regularization parameters in inverse problems are optimally chosen by the fundamental principle of balancing approximation and sampling errors. We use this principle to balance learning errors justified rule for the choice of the regularization parameter. In contrast to the state of the art, our approach allows source and target distributions with disjoint supports. An empirical comparative study on benchmark datasets underpins the performance of our approach.

我们解决了在无监督领域适应中选择合理正则化参数的未解决算法设计问题。这个问题引人注目，因为目标领域没有可用的标签。我们的方法始于观察到，广泛使用的通过源特征表示和目标特征表示之间的距离度量来惩罚源误差最小化的方法，与正则化病态逆问题具有相似特征。逆问题中的正则化参数是通过平衡近似误差和采样误差的基本原则来最佳选择的。我们利用这一原则来平衡学习误差，从而为正则化参数的选择提供合理依据。与现有技术相比，我们的方法允许源分布和目标分布具有不相交的支持。对基准数据集的实证比较研究支持了我们方法的性能。

Domain adaptation uses the knowledge in a source domain to improve the performance of an algorithm on a related target domain [1]. In particular, domain adaptation tackles domain shifts in machine learning applications: Medical diagnostic systems should be adapted to new physical human variations; Industrial quality inspection systems should be accurate for new products; Self-driving cars should be able to adapt to new geographical environments and weather conditions. In this work, we focus on unsupervised domain adaptation where labels are only available in the source domain.

领域适应利用源领域中的知识来提高算法在相关目标领域的性能 [1]。特别是，领域适应解决机器学习应用中的领域转移问题：医疗诊断系统应适应新的身体人类变异；工业质量检查系统应对新产品保持准确；自动驾驶汽车应能够适应新的地理环境和气候条件。在这项工作中，我们专注于无监督领域适应，其中标签仅在源领域中可用。

There are mainly two types of approaches for unsupervised domain adaptation: importance weighting

[2, 3, 4, 5, 6, 7]

and feature representation learning

[8, 9, 10, 11, 12, 13]

. In this work,we focus on feature representation learning which goes beyond classical importance weighting by allowing a target distribution with support outside of the source distribution. The core idea behind feature representation learning approaches is to map the data into a new feature space where the source and target

无监督领域适应主要有两种方法：重要性加权

[2, 3, 4, 5, 6, 7]

和特征表示学习

[8, 9, 10, 11, 12, 13]

。在这项工作中，我们专注于特征表示学习，它超越了经典的重要性加权，允许目标分布的支持超出源分布。特征表示学习方法的核心思想是将数据映射到一个新的特征空间，在这个空间中，源和目标

35th Conference on Neural Information Processing Systems (NeurIPS 2021). data representations appear similar, and where enough information is preserved for prediction [14]. The similarity is often realized by regularization using distance measures between source and target representations

[15, 16, 17, 18, 19, 20, 21]

. However,the performance of such methods crucially depends on the choice of the regularization parameter which penalizes the distance. The problem we investigate in this work is to choose this parameter, which is sophisticated without any target labels.

第35届神经信息处理系统会议（NeurIPS 2021）。数据表示看起来相似，并且保留了足够的信息用于预测 [14]。这种相似性通常通过使用源表示和目标表示之间的距离度量进行正则化来实现

[15, 16, 17, 18, 19, 20, 21]

。然而，这种方法的性能在很大程度上依赖于正则化参数的选择，该参数对距离进行惩罚。我们在本研究中探讨的问题是如何选择这个参数，而没有任何目标标签的情况下，这一过程是复杂的。

Figure 1: Unsupervised domain adaptation on Transformed Moons. Left: Target data (black dots) partially outside of the support of the source data (blue +,orange

\times

). The common assumption of bounded density ratio is violated in large regions. In contrast, all our assumptions are satisfied. Our method identifies the best parameter of a domain adaptation algorithm [16] (green solid) which improves training on source data only (red dashed). Right: Regularization parameter (x-axis) which penalizes a distance [25] (purple) leading to models with different source error (blue) and target error (black). Importance weighted validation (IWV) shows the smallest error (red) for models without domain adaptation

(α_{IWV} = 0)

. In contrast,our approach identifies the optimal parameter

(α_{BP} = 1)

图1：在变换月亮数据集上的无监督领域适应。左侧：目标数据（黑点）部分位于源数据（蓝色+，橙色

\times

）的支持范围之外。大区域内违反了有界密度比的共同假设。相反，我们的所有假设都得到了满足。我们的方法识别出领域适应算法的最佳参数 [16]（绿色实线），该参数仅改善源数据上的训练（红色虚线）。右侧：正则化参数（x轴），对距离进行惩罚 [25]（紫色），导致不同源误差（蓝色）和目标误差（黑色）的模型。重要性加权验证（IWV）显示无领域适应模型的最小误差（红色）

(α_{IWV} = 0)

。相反，我们的方法识别出最佳参数

(α_{BP} = 1)

。

While remarkable theoretical results have been achieved which quantify the generalization ability of domain adaptation models

[8, 22, 19, 20, 21]

,the choice of the regularization parameter which is crucial for finding such models has not systematically been addressed. Even though some parameter choice strategies exist, they are either purely heuristically driven or very limited by their assumptions [23]. Typical approaches are fixing the regularization parameters [12], minimizing the source error [16], balancing the source error and a distance [17], multiplying a fixed weighting parameter (e.g. 1 in [16]) by a heuristic schedule value that increases during training, or, (importance) weighting the input samples by the ratio between target and source density [3, 24, 23]. One common problem shared among all these approaches is that they all can fail if the density ratio is unbounded. Such unbounded density ratio is typical for many of the high dimensional problems considered in machine learning [19], e.g. see Figure 1. Besides the aforementioned issues, the lack of principled strategies for parameter choice causes misinterpretations in the ranking of domain adaptation methods which are traditionally compared by performance, while often relying on different parameter choice strategies.

尽管已经取得了显著的理论结果，量化了领域适应模型的泛化能力

[8, 22, 19, 20, 21]

，但对于寻找此类模型至关重要的正则化参数的选择尚未系统地进行探讨。尽管存在一些参数选择策略，但它们要么完全基于启发式，要么受到其假设的限制 [23]。典型的方法包括固定正则化参数 [12]、最小化源误差 [16]、平衡源误差和距离 [17]、将固定权重参数（例如 [16] 中的 1）乘以在训练过程中增加的启发式调度值，或者通过目标和源密度之间的比率对输入样本进行（重要性）加权 [3, 24, 23]。所有这些方法共同面临的一个常见问题是，如果密度比是无界的，它们都可能失败。这种无界密度比在机器学习中考虑的许多高维问题中是典型的 [19]，例如见图 1。除了上述问题外，缺乏原则性参数选择策略导致了领域适应方法排名的误解，这些方法通常通过性能进行比较，而往往依赖于不同的参数选择策略。

In this work, we propose a principled method for choosing distance-penalizing parameters of feature representation learning approaches for unsupervised domain adaptation. Our approach starts with the observation that the distance-regularization setting of domain adaptation shares characteristics with regularized ill-posed inverse problems (see Table 1). In inverse problems, the regularization parameter can be optimally chosen by the fundamental balancing principle which optimizes an approximation-sample (bias-variance) trade-off

[26, 27, 28]

. We apply this principle for balancing domain distance and learning errors of target error bounds. In particular, we approach the problem of non-computable terms in the target error bound by a new algorithmic criterion for approximating the value of balance. We call our method the Balancing Principle for Domain Adaptation (BPDA).

在本研究中，我们提出了一种原则性的方法，用于选择无监督领域适应中特征表示学习方法的距离惩罚参数。我们的方法始于观察领域适应的距离正则化设置与正则化病态逆问题的特征相似（见表 1）。在逆问题中，正则化参数可以通过基本的平衡原则进行最佳选择，该原则优化了近似样本（偏差-方差）权衡

[26, 27, 28]

。我们应用这一原则来平衡领域距离和目标误差界限的学习误差。特别地，我们通过一种新的算法标准来处理目标误差界限中不可计算项的问题，以近似平衡值。我们将我们的方法称为领域适应的平衡原则（BPDA）。

The BPDA is general in the sense that it can be applied based on different target error bounds, e.g. on

[8, 22, 19, 20, 21]

. To the best of our knowledge,the BPDA is the first principled method for parameter choice in unsupervised domain adaptation that allows an unbounded ratio between target and source density. We provide a bound on the generalization error of the best model corresponding to the parameter chosen by the BPDA. Finally, we empirically investigate the behavior of the BPDA based on two target error bounds, different domain adaptation methods and benchmark datasets. Our results show that the BPDA outperforms or is on par with the state of the art on the problem of choosing the regularization parameter, on several domain adaptation methods; applied on different

BPDA 在某种意义上是通用的，因为它可以基于不同的目标误差界限进行应用，例如在

[8, 22, 19, 20, 21]

上。根据我们所知，BPDA 是第一个在无监督领域适应中用于参数选择的原则性方法，它允许目标密度与源密度之间存在无限制的比率。我们提供了与 BPDA 选择的参数对应的最佳模型的泛化误差界限。最后，我们基于两个目标误差界限、不同的领域适应方法和基准数据集，对 BPDA 的行为进行了实证研究。我们的结果表明，BPDA 在选择正则化参数的问题上，在几种领域适应方法中，表现优于或与最先进的技术相当；并应用于不同的

Notation Let

X \subset R^{n}

be an input space and

Y

be a discrete label space. Following the classical setting of unsupervised domain adaptation [25],we consider two datasets: A source dataset

(x, y) =

((x_{1}, l_{S} (x_{1})), \dots, (x_{s}, l_{S} (x_{s}))) \in {(X \times Y)}^{s}

with inputs

x_{1}, \dots, x_{s}

independently drawn according to some source distribution (Borel probability measure)

p_{S}

X

and labeled according to some labeling function

^{1} l_{S} : X \to Y

,and,an unlabeled target dataset

x^{'} = (x_{1}^{'}, \dots, x_{t}^{'}) \in X^{t}

with elements independently drawn according to some target distribution

p_{T}

X

. Throughout this work, we focus on loss functions

L : Y \times Y \to [0, \infty)

which satisfy

L (y, y) = 0

. For example consider the 0-1 loss

L (y_{1}, y_{2}) := 1 [y_{1} \neq y_{2}]

,where

1 [P]

is 1 iff the predicate

P

is true and 0 otherwise,and the quadratic loss function

L (y_{1}, y_{2}) := {| y_{1} - y_{2} |}^{2}

. We denote the source error by

ε_{S} (f) = ε_{S} (f, l_{S})

with cross-error defined as

ε_{S} (f, g) := E_{x \sim p_{S}} [L (f (x), g (x))]

and its empirical sample estimate by

{\hat{ε}}_{S} (f) = {\hat{ε}}_{S} (f, l_{S})

with

{\hat{ε}}_{S} (f, g) := \sum_{i = 1}^{s} L (f (x_{i}), g (x_{i}))

. We denote the analogously defined target error by

ε_{T} (f)

,target cross-error by

ε_{T} (f, g)

and its empirical sample estimate by

{\hat{ε}}_{T} (f)

with empirical cross-error

{\hat{ε}}_{T} (f, g)

. Throughout this work,we focus on target cross-errors

ε_{T} (f, g)

which satisfy the triangle inequality.

记号设

X \subset R^{n}

为输入空间，

Y

为离散标签空间。遵循无监督领域适应的经典设置 [25]，我们考虑两个数据集：一个源数据集

(x, y) =

((x_{1}, l_{S} (x_{1})), \dots, (x_{s}, l_{S} (x_{s}))) \in {(X \times Y)}^{s}

，其输入

x_{1}, \dots, x_{s}

独立地根据某些源分布（博雷尔概率测度）

p_{S}

在

X

上抽取，并根据某些标记函数

^{1} l_{S} : X \to Y

进行标记；另一个是未标记的目标数据集

x^{'} = (x_{1}^{'}, \dots, x_{t}^{'}) \in X^{t}

，其元素独立地根据某些目标分布

p_{T}

在

X

上抽取。在本研究中，我们专注于满足

L (y, y) = 0

的损失函数

L : Y \times Y \to [0, \infty)

。例如，考虑 0-1 损失

L (y_{1}, y_{2}) := 1 [y_{1} \neq y_{2}]

，其中

1 [P]

为 1 当且仅当谓词

P

为真时，其他情况下为 0，以及二次损失函数

L (y_{1}, y_{2}) := {| y_{1} - y_{2} |}^{2}

。我们用

ε_{S} (f) = ε_{S} (f, l_{S})

表示源误差，交叉误差定义为

ε_{S} (f, g) := E_{x \sim p_{S}} [L (f (x), g (x))]

，其经验样本估计为

{\hat{ε}}_{S} (f) = {\hat{ε}}_{S} (f, l_{S})

，并且

{\hat{ε}}_{S} (f, g) := \sum_{i = 1}^{s} L (f (x_{i}), g (x_{i}))

。我们用

ε_{T} (f)

表示类似定义的目标误差，目标交叉误差用

ε_{T} (f, g)

表示，其经验样本估计为

{\hat{ε}}_{T} (f)

，经验交叉误差为

{\hat{ε}}_{T} (f, g)

。在本研究中，我们专注于满足三角不等式的目标交叉误差

ε_{T} (f, g)

。

Learning setup In this work, we focus on feature representation learning algorithms for domain adaptation. These approaches aim at finding two learning models: A representation mapping

ϕ \in

Φ \subset {ϕ : X \to R}

into some representation space

R \subset R^{m}

and a classifier

g \in G \subset {g : R \to Y}

. Loosely speaking,the aim is to find a mapping

ϕ

under which the source representations

ϕ (x) :=

(ϕ (x_{1}), \dots, ϕ (x_{s}))

and the target representations

ϕ (x^{'}) := (ϕ (x_{1}^{'}), \dots, ϕ (x_{t}^{'}))

appear similar,and, at the same time,enough information is preserved for prediction [14] by

g (x)

. A common approach to realize this aim is to solve the following objective function [19]

学习设置在本研究中，我们专注于用于领域适应的特征表示学习算法。这些方法旨在找到两个学习模型：一个表示映射

ϕ \in

Φ \subset {ϕ : X \to R}

到某个表示空间

R \subset R^{m}

和一个分类器

g \in G \subset {g : R \to Y}

。宽泛地说，目标是找到一个映射

ϕ

，在该映射下，源表示

ϕ (x) :=

(ϕ (x_{1}), \dots, ϕ (x_{s}))

和目标表示

ϕ (x^{'}) := (ϕ (x_{1}^{'}), \dots, ϕ (x_{t}^{'}))

看起来相似，同时保留足够的信息以进行预测 [14] 通过

g (x)

。实现这一目标的常见方法是解决以下目标函数 [19]。

where

d

is a distance measure between source and target representations and

α \in [0, \infty)

is a parameter

^{2}

. Good choices for

d

in Eq. (1) have been identified to be the Wasserstein distance [29,30], the Maximum Mean Discrepancy [31, 32], moment distances [17, 33, 18, 34, 35, 36], adversarially learned distances [16, 37] and other measures of divergence [38, 39, 19, 20].

其中

d

是源表示和目标表示之间的距离度量，

α \in [0, \infty)

是一个参数

^{2}

。在方程 (1) 中，已识别出

d

的良好选择为 Wasserstein 距离 [29,30]、最大均值差异 [31, 32]、矩距离 [17, 33, 18, 34, 35, 36]、对抗学习的距离 [16, 37] 和其他发散度量 [38, 39, 19, 20]。

Problem For some

α \in [0, \infty)

,let

g_{α} \circ ϕ_{α}

denote the minimizer of Eq. (1). Given an increasing sequence of parameters

α_{1}, \dots, α_{w} \in [0, \infty)

with

α_{1} = 0

,the problem studied in this work is to choose the parameter

α

in the sequence

α_{1}, \dots, α_{w}

with the lowest target error

ε_{T} (g_{α} \circ ϕ_{α})

问题对于某些

α \in [0, \infty)

，令

g_{α} \circ ϕ_{α}

表示方程 (1) 的最小化器。给定一个递增的参数序列

α_{1}, \dots, α_{w} \in [0, \infty)

，其中

α_{1} = 0

，本研究中研究的问题是选择序列

α_{1}, \dots, α_{w}

中具有最低目标误差

ε_{T} (g_{α} \circ ϕ_{α})

的参数

α

。

where

D (α)

gives a notion of domain distance (cf. [25]) by quantifying a distance between source and target data representations and

E (α)

comprises different learning errors. We assume that

E (α)

is bounded by some constant

B > 0

. The general form in Eq. (2) is satisfied by many error bounds

[8, 22, 20, 21, 34]

which all can be taken as a basis for our approach (more detailed examples are provided in Section 3 and Section 4). One problem that complicates the minimization of these That is,

E (α)

cannot be directly estimated. The BPDA overcomes this problem by a new criterion for estimating the value of balance between the normalized terms

E (α)

and

D (α)

. The BPDA is detailed in Algorithm 1.

在这里，

D (α)

给出了领域距离的概念（参见 [25]），通过量化源数据和目标数据表示之间的距离，而

E (α)

包含不同的学习误差。我们假设

E (α)

被某个常数

B > 0

所限制。方程 (2) 中的一般形式满足许多误差界限

[8, 22, 20, 21, 34]

，这些误差界限都可以作为我们方法的基础（更详细的例子在第 3 节和第 4 节中提供）。一个复杂化这些最小化的问题是

E (α)

不能被直接估计。BPDA 通过一个新的标准来克服这个问题，以估计归一化项

E (α)

和

D (α)

之间的平衡值。BPDA 在算法 1 中进行了详细描述。

The learning model $g_{α_{BP}} \circ ϕ_{α_{BP}}$ identified by the BPDA satisfies a generalization bound, see Section 4. If the learning errors term $E (α)$ is non-decreasing,then the target error of $g_{α_{BP}} \circ ϕ_{α_{BP}}$ is only a constant factor away from the minimum $min_{α \in [0, \infty)} D (α) + E (α)$ of the instantiation bound in Eq. (2).

BPDA 识别的学习模型 $g_{α_{BP}} \circ ϕ_{α_{BP}}$ 满足泛化界限，参见第 4 节。如果学习误差项 $E (α)$ 是非递减的，则 $g_{α_{BP}} \circ ϕ_{α_{BP}}$ 的目标误差仅与方程 (2) 中实例化界限的最小值 $min_{α \in [0, \infty)} D (α) + E (α)$ 相差一个常数因子。

Related work Approaches which follow Eq. (1) are sometimes interpreted as learning domain-invariant representation. Note that the minimization of

d (ϕ (x), ϕ (x^{'}))

in Eq. (1) to achieve

ϕ (x) =

ϕ (x^{'})

differs from the conception as regularization

[40, 41]

. In fact,minimization of the distance means unjustified over-penalization which might lead to deteriorated performance [19]. Interestingly, our interpretation as regularization problem opens up a powerful toolbox of mathematical techniques. Our approach takes up the technique of balancing stability and approximation in regularized ill-posed inverse problems. The balancing principle has its origins in [26] devoted to nonparametric regression estimation and has been introduced in the context of ill-posed problems [27] and in supervised learning with kernels [42]. Following this line of research, we propose to apply the mathematical techniques underlying the balancing principle in the context of domain adaptation. The most related principled parameter choice methods in the context of unsupervised domain adaptation are importance-weighted cross-validation [3] and its extensions [24,23]. In contrast to these methods, our method is not restricted by the assumption of a bounded ratio between target and source density. One empirically driven method which is related to ours is

[17, 43]

which aims at balancing the source error and a distance between source and target data representations. However, this method is not theoretically justified as it ignores the minimal combined error of a classifier on representations as defined in [8]. Nevertheless, if the minimal combined error is negligibly small, our results provide a theoretical explanation of the principles underlying the success of [17,43]. Our method relies on bounds on the target error such as e.g.

[8, 22, 19, 20, 21, 34]

相关工作遵循方程 (1) 的方法有时被解释为学习领域不变表示。注意，在方程 (1) 中最小化

d (ϕ (x), ϕ (x^{'}))

以实现

ϕ (x) =

ϕ (x^{'})

与作为正则化的概念

[40, 41]

是不同的。实际上，距离的最小化意味着不合理的过度惩罚，这可能导致性能下降 [19]。有趣的是，我们将其解释为正则化问题，打开了一套强大的数学技术工具箱。我们的方法采用了在正则化病态逆问题中平衡稳定性和近似性的技术。平衡原则源于 [26]，专注于非参数回归估计，并在病态问题的背景下引入 [27]，以及在使用核的监督学习中 [42]。沿着这条研究路线，我们建议在领域适应的背景下应用平衡原则所依据的数学技术。在无监督领域适应的背景下，最相关的原则性参数选择方法是重要性加权交叉验证 [3] 及其扩展 [24,23]。与这些方法相比，我们的方法不受目标和源密度之间有界比率假设的限制。一种与我们的方法相关的经验驱动方法是

[17, 43]

，旨在平衡源误差和源与目标数据表示之间的距离。然而，由于忽略了分类器在表示上的最小组合误差 [8]，该方法在理论上并不成立。尽管如此，如果最小组合误差微不足道，我们的结果为 [17,43] 成功背后的原则提供了理论解释。我们的方法依赖于对目标误差的界限，例如

[8, 22, 19, 20, 21, 34]

。

is the empirical

G △ G

-divergence

[44, 25]

and

λ_{G} (ϕ) := inf_{f \in G} (ε_{S} (f \circ ϕ) + ε_{T} (f \circ ϕ))

is the minimum possible combined error determined by the application of

G

. Using Eq. (3),the source error can be further upper bounded by the empirical source error. However,the term

λ_{G} (ϕ)

cannot be estimated based on given datasets as it depends on the unknown labeling function

l_{T}

是经验

G △ G

-散度

[44, 25]

，而

λ_{G} (ϕ) := inf_{f \in G} (ε_{S} (f \circ ϕ) + ε_{T} (f \circ ϕ))

是通过应用

G

确定的最小可能的综合错误。使用公式 (3)，源错误可以进一步由经验源错误上界。然而，项

λ_{G} (ϕ)

不能基于给定的数据集进行估计，因为它依赖于未知的标记函数

l_{T}

。

In [20],the bound in Eq. (4) is generalized to multiple classes

Y := {1, \dots, k}

and scoring functions

H \subset {h : R \to R^{k}}

,where the output on each dimension indicates the confidence of prediction. For some

h \in H

,let us denote by

g_{h} : R \to Y, x \mapsto \arg max_{i \in {1, \dots, k}} h^{(i)} (ϕ (x))

with

h^{(i)} (z)

being the

i

-th component of

h (z)

. Let further

ε_{T}

denote the target error based on 0-1 loss. Then,with probability at least

1 - δ

在 [20] 中，公式 (4) 中的界限被推广到多个类别

Y := {1, \dots, k}

和评分函数

H \subset {h : R \to R^{k}}

，其中每个维度的输出表示预测的置信度。对于某些

h \in H

，我们用

g_{h} : R \to Y, x \mapsto \arg max_{i \in {1, \dots, k}} h^{(i)} (ϕ (x))

表示，其中

h^{(i)} (z)

是

h (z)

的第

i

个分量。进一步地，令

ε_{T}

表示基于 0-1 损失的目标错误。那么，概率至少为

1 - δ

where

ε_{S}^{(ρ)} (h) := \int_{X} Λ_{ρ} \circ ρ_{h} (x, l_{S}) d p_{S} (x)

with

ρ_{h} (x, l_{S}) := 1 / 2 (h^{(l_{S} (x))} (x) - max_{y \neq l_{S} (x)} h^{(y)} (x))

and

Λ_{ρ} (x)

being

1 - x / ρ

0 \leq x < ρ, 0

ρ \leq x

and 1 otherwise. Further,the target terms

T, p_{T}, l_{T}

are defined analogously to

S, p_{S}, l_{S}

,respectively; the combined error is defined by

λ_{H} (ϕ) :=

inf_{h \in H} (ε_{S}^{(ρ)} (h \circ ϕ) + ε_{T}^{(ρ)} (h))

,and the empirical margin disparity discrepancy (MDD) is given by

其中

ε_{S}^{(ρ)} (h) := \int_{X} Λ_{ρ} \circ ρ_{h} (x, l_{S}) d p_{S} (x)

，当

ρ_{h} (x, l_{S}) := 1 / 2 (h^{(l_{S} (x))} (x) - max_{y \neq l_{S} (x)} h^{(y)} (x))

和

Λ_{ρ} (x)

为

1 - x / ρ

时，如果

0 \leq x < ρ, 0

为

ρ \leq x

，否则为 1。此外，目标项

T, p_{T}, l_{T}

类似于

S, p_{S}, l_{S}

定义；综合错误由

λ_{H} (ϕ) :=

inf_{h \in H} (ε_{S}^{(ρ)} (h \circ ϕ) + ε_{T}^{(ρ)} (h))

定义，经验边际差异不一致性 (MDD) 由以下公式给出：

Balancing principle for regularized inverse problems Let

H

and

K

be two Hilbert spaces and is to find some function

f

satisfying

V f = g

,see e.g. [45,46,40,41] and references therein. In general, the above problem is ill-posed, i.e. a solution does either not exist, is not unique or does not depend continuously on

g

. Existence and uniqueness can be approached by using the the following minimizer as approximation of

f

正则化逆问题的平衡原则设

H

和

K

为两个希尔伯特空间，目标是找到某个函数

f

，使其满足

V f = g

，参见例如 [45,46,40,41] 及其中的参考文献。一般而言，上述问题是病态的，即解要么不存在，要么不唯一，或者不连续依赖于

g

。可以通过使用以下最小化器作为

f

的近似来接近存在性和唯一性。

where

S (α)

is called sampling error made by considering noisy approximations

\hat{V}

and

\hat{g}

V

and

g

,respectively,and,

A (α)

is called approximation error originating from adding the regularizer

∥ f ∥_{H}^{2}

. Commonly,

S (α)

decreases while

A (α)

increases with increasing

α

. We refer to [42] for detailed examples. The fact that

S (α)

decreases and

A (α)

increases motivates the so-called balancing principle which aims at computing

α^{*}

such that

S (α^{*}) = A (α^{*}) [26, 27, 42, 41, 28]

. As a result,the balancing principle provides a procedure for approximating

α^{*}

without having access to the values of

A (α)

. The balancing principle obtains optimal error rates in many settings [47,48,49,42,41,28].

其中

S (α)

被称为考虑到噪声近似

\hat{V}

和

\hat{g}

的采样误差，分别对应于

V

和

g

，而

A (α)

被称为由于添加正则化项

∥ f ∥_{H}^{2}

而产生的近似误差。通常，随着

α

的增加，

S (α)

会减少，而

A (α)

会增加。我们参考 [42] 以获取详细示例。

S (α)

减少和

A (α)

增加的事实激励了所谓的平衡原则，旨在计算

α^{*}

使得

S (α^{*}) = A (α^{*}) [26, 27, 42, 41, 28]

。因此，平衡原则提供了一种在没有访问

A (α)

值的情况下近似

α^{*}

的程序。平衡原则在许多设置中获得了最佳误差率 [47,48,49,42,41,28]。

In the following, we present the mathematical foundations of Algorithm 1 in two steps. In a first step, in Subsection 4.1, we state our assumptions and detail our idea of choosing the value of balance between terms in a target error bound. In a second step, in Subsection 4.2, we propose an algorithmic criterion, the balancing principle estimate for approximating the theoretical choice of the balancing value without target labels. Based on this criterion, we explain why we expect Algorithm 1 to be accurate.

在接下来的内容中，我们将算法 1 的数学基础分为两个步骤进行介绍。在第一步中，在小节 4.1 中，我们陈述我们的假设，并详细说明我们选择目标误差界限中各项之间平衡值的想法。在第二步中，在小节 4.2 中，我们提出了一种算法标准，即平衡原则估计，用于在没有目标标签的情况下近似理论上平衡值的选择。基于这一标准，我们解释了为什么我们期望算法 1 是准确的。

Assumptions In the following,let

g_{α} \circ ϕ_{α}

be a minimizer of Eq. (1) and assume that a target error bound holds which is of the form given by Eq. (2). Our approach is based on the plausible assumptions (a) that the function

α \mapsto E (α)

is continuous and bounded by some constant

B > 0

, and (b) that the function

α \mapsto D (α)

is continuous,non-increasing and non-degenerate,i.e.

D (0) > 0

. We further make the technical assumption (c) that

lim_{α \to \infty} D (α) / D (0) < sup_{α \in [0, \infty)} E (α) / B

. Note that (c) is satisfied in most standard cases,where

D (α) \to 0

for

α, t \to \infty

. For example consider the two accompanying target error bounds in Section 3 as discussed at the end of this section.

假设在接下来的内容中，设

g_{α} \circ ϕ_{α}

为方程 (1) 的最小化器，并假设存在一个形式如方程 (2) 所示的目标误差界限。我们的方法基于以下合理假设：(a) 函数

α \mapsto E (α)

是连续的，并且被某个常数

B > 0

有界；(b) 函数

α \mapsto D (α)

是连续的、非递增的且非退化的，即

D (0) > 0

。我们进一步做出技术假设 (c)，即

lim_{α \to \infty} D (α) / D (0) < sup_{α \in [0, \infty)} E (α) / B

。请注意，(c) 在大多数标准情况下是满足的，其中

D (α) \to 0

对于

α, t \to \infty

。例如，考虑本节末尾讨论的第 3 节中的两个伴随目标误差界限。

Bridging regularized inverse problems and domain adaptation Under the assumptions above, our domain adaptation learning setup in Section 2 shares characteristics with the setting of regularized inverse problems as described in Section 3. Indeed,the Tikhonov regularizer

∥ f ∥_{H}^{2}

in Eq. (8) is similarly applied as the distance-regularizer

d (ϕ (x), ϕ (x^{'}))

in Eq. (1). In addition,error bounds for inverse problems as given by Eq. (9) show a similar form as target error bounds following Eq. (2). The sampling error

S (α)

in Eq. (9) decreases similarly to the domain distance in Eq. (2) and the approximation error

A (α)

in Eq. (9) cannot be estimated similarly to the learning errors

E (α)

in Eq. (2). However,in the domain adaptation setting,the term

E (α)

does not necessarily increase. We approach this issue by considering the least non-decreasing majorant of

E (α)

桥接正则化逆问题与领域适应在上述假设下，我们在第2节中的领域适应学习设置与第3节中描述的正则化逆问题的设置具有相似特征。实际上，方程（8）中的Tikhonov正则化器

∥ f ∥_{H}^{2}

与方程（1）中的距离正则化器

d (ϕ (x), ϕ (x^{'}))

的应用方式相似。此外，方程（9）给出的逆问题的误差界限与方程（2）后的目标误差界限形式相似。方程（9）中的采样误差

S (α)

以与方程（2）中的领域距离相似的方式减少，而方程（9）中的近似误差

A (α)

不能以与方程（2）中的学习误差

E (α)

相似的方式进行估计。然而，在领域适应设置中，项

E (α)

不一定会增加。我们通过考虑

E (α)

的最小非递减上界来解决这个问题。

Tikhonov-regularized inverse problem	Distance-regularized domain adaptation
$f_{α} \in \arg min ∥ \hat{V} f - \hat{g} ∥_{K}^{2} + α ∥ f ∥_{H}^{2}$ $f \in H$	$g_{α} \circ ϕ_{α} \in \arg min {\hat{ε}}_{S} (g \circ ϕ) + α d (ϕ (x), ϕ (x^{'}))$ $g \in G, ϕ \in Φ$
${‖ \begin{matrix} f_{α} - f_{H} \end{matrix} ‖}_{H} \leq S (α) + A (α)$	$ε_{T} (g_{α} \circ ϕ_{α}) \leq D (α) + E (α)$
decreasing sampling error $S (α)$	decreasing domain distance $D (α)$
increasing approximation error $A (α)$	bounded learning errors $E (α)$
$A (α)$ not estimable	$E (α)$ not estimable
balance $A (α^{}) = S (α^{})$	balance $\frac{D (α^{})}{D (0)} = \frac{\bar{E} (α^{})}{B}$

Tikhonov 正则化逆问题	距离正则化领域适应
$f_{α} \in \arg min ∥ \hat{V} f - \hat{g} ∥_{K}^{2} + α ∥ f ∥_{H}^{2}$ $f \in H$	$g_{α} \circ ϕ_{α} \in \arg min {\hat{ε}}_{S} (g \circ ϕ) + α d (ϕ (x), ϕ (x^{'}))$ $g \in G, ϕ \in Φ$
${‖ \begin{matrix} f_{α} - f_{H} \end{matrix} ‖}_{H} \leq S (α) + A (α)$	$ε_{T} (g_{α} \circ ϕ_{α}) \leq D (α) + E (α)$
降低采样误差 $S (α)$	降低领域距离 $D (α)$
增加近似误差 $A (α)$	有界学习误差 $E (α)$
$A (α)$ 不可估计	$E (α)$ 不可估计
平衡 $A (α^{}) = S (α^{})$	平衡 $\frac{D (α^{})}{D (0)} = \frac{\bar{E} (α^{})}{B}$

Figure 2: Left: The BPDA in Algorithm 1 overcomes the problem of the unknown learning errors term

E (α)

by approximating

α^{*}

which balances

D (α) / D (0)

(green) and the least non-decreasing majorant

\bar{E} (α) / B

(red dashed) of

E (α) / B

(red). Right: Average and standard deviation over 10 repetitions of estimated learning errors

E

and the domain distance

D

of the accompanying target error bound Eq. (4) for the models computed by Eq. (1) with the Maximum Mean Discrepancy [51] as distance. The BPDA chooses the value

α_{BP} = 10^{- 2}

near the estimated balancing value.

图 2：左：算法 1 中的 BPDA 通过近似

α^{*}

来克服未知学习误差项

E (α)

的问题，该近似平衡了

D (α) / D (0)

（绿色）和

E (α) / B

（红色）的最小非递减主导函数

\bar{E} (α) / B

（红色虚线）。右：在 10 次重复中估计的学习误差

E

和伴随目标误差界限 Eq. (4) 的领域距离

D

的平均值和标准差，对于通过 Eq. (1) 计算的模型，使用最大均值差异 [51] 作为距离。BPDA 选择接近估计平衡值的值

α_{BP} = 10^{- 2}

。

assumptions (a)-(c) above. See Figure 2 for an illustration. Algorithm 1 approximates

α^{*}

. If

E (α)

is non-decreasing and the bound in Eq. (10) holds with equality, then the rate of the target error

ε_{T} (g_{α^{*}} \circ ϕ_{α^{*}})

is optimal,i.e.

ε_{T} (g_{α^{*}} \circ ϕ_{α^{*}})

is only a constant factor away from the optimum

inf_{α \in [0, \infty)} ε_{T} (g_{α} \circ ϕ_{α})

. See the supplementary material for a proof. This optimality property is shared with related regularization settings [47,48,49,42,28].

上述假设 (a)-(c)。请参见图 2 以获取说明。算法 1 近似

α^{*}

。如果

E (α)

是非递减的，并且 Eq. (10) 中的界限成立，则目标误差

ε_{T} (g_{α^{*}} \circ ϕ_{α^{*}})

的速率是最优的，即

ε_{T} (g_{α^{*}} \circ ϕ_{α^{*}})

仅与最优值

inf_{α \in [0, \infty)} ε_{T} (g_{α} \circ ϕ_{α})

相差一个常数因子。有关证明，请参见补充材料。该最优性属性与相关的正则化设置 [47,48,49,42,28] 共享。

Two accompanying target error bounds Let us now discuss the reasoning above based on the two error bounds described in Section 3. First, recall the target error bound of [25] in Eq. (4). If we take

D (α) := {\hat{d}}_{G △ G} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{t . G . δ}

and

E (α) := ε_{S} (g_{α} \circ ϕ_{α}) + λ_{G} (ϕ_{α})

,it is natural to assume

D (α)

to decrease with

α

,especially for adversarial approaches which penalize the empirical

G △ G

-divergence

{\hat{d}}_{G △ G} (ϕ (x), ϕ (x^{'}))

,see [16]. It also holds that

E (α) \leq 3

for all

α \in [0, \infty)

. For the balancing value to exist for large sample size

t

,we need to verify that

D (α) / D (0) \to 0

for

t, α \to \infty

. However,this is satisfied for most function classes

Φ

,since it can be assumed that the constant function

ϕ : x \mapsto c \in R

which achieves

{\hat{d}}_{G △ G} (ϕ (x), ϕ (x^{'})) = 0

is contained in

Φ

. That is,

{\hat{d}}_{G △ G} (ϕ (x), ϕ (x^{'})) = 0

can be achieved for

α \to \infty

and consequently

D (α) / D (0) \to 0

for

t, α \to \infty

. Consider now the bound in Eq. (6). By using

D (α) := {\hat{d}}_{h, H}^{(ρ)} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{s + t, H, δ}, E (α) :=

ε_{S}^{(ρ)} (h \circ ϕ) + λ_{H} (ϕ)

and

B = 3

,all assumptions above are naturally satisfied similarly to the target error bound in Eq. (4) and the balancing value exists. In Section 5, we empirically investigate the performance of our method based on our two accompanying target error bounds. However, let us first show how we can overcome the problem of the non-computable term

E (α)

两个伴随的目标误差界限现在让我们讨论基于第3节中描述的两个误差界限的推理。首先，回顾[25]中在公式(4)中的目标误差界限。如果我们取

D (α) := {\hat{d}}_{G △ G} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{t . G . δ}

和

E (α) := ε_{S} (g_{α} \circ ϕ_{α}) + λ_{G} (ϕ_{α})

，那么自然假设

D (α)

会随着

α

的增加而减少，特别是对于惩罚经验

G △ G

-散度

{\hat{d}}_{G △ G} (ϕ (x), ϕ (x^{'}))

的对抗性方法，见[16]。对于所有

α \in [0, \infty)

，也成立

E (α) \leq 3

。为了在大样本量

t

下存在平衡值，我们需要验证

D (α) / D (0) \to 0

对于

t, α \to \infty

。然而，对于大多数函数类

Φ

，这一点是满足的，因为可以假设常数函数

ϕ : x \mapsto c \in R

能够达到

{\hat{d}}_{G △ G} (ϕ (x), ϕ (x^{'})) = 0

，并且包含在

Φ

中。也就是说，

{\hat{d}}_{G △ G} (ϕ (x), ϕ (x^{'})) = 0

可以通过

α \to \infty

实现，因此

D (α) / D (0) \to 0

对于

t, α \to \infty

也是成立的。现在考虑公式(6)中的界限。通过使用

D (α) := {\hat{d}}_{h, H}^{(ρ)} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{s + t, H, δ}, E (α) :=

ε_{S}^{(ρ)} (h \circ ϕ) + λ_{H} (ϕ)

和

B = 3

，上述所有假设自然满足，类似于公式(4)中的目标误差界限，并且存在平衡值。在第5节中，我们基于我们的两个伴随目标误差界限实证研究我们方法的性能。然而，让我们首先展示如何克服不可计算项

E (α)

的问题。

The balancing principle estimate in Definition 2 is based on checking an upper bound on the empirical cross-error

{\hat{ε}}_{T} (f_{i}, f_{j})

between two models

f_{i}, f_{j}

resulting from two different values

α_{i}, α_{j}

of the regularization parameter, respectively. The empirical cross-error does not contain information about the unknown target labels

l_{T} (x_{1}^{'}), \dots, l_{T} (x_{t}^{'})

and can be computed based on available data. The Eq. (11) can be explained as follows.

定义 2 中的平衡原则估计基于对两个模型

f_{i}, f_{j}

之间的经验交叉误差

{\hat{ε}}_{T} (f_{i}, f_{j})

的上界进行检查，这两个模型分别来自正则化参数的两个不同值

α_{i}, α_{j}

。经验交叉误差不包含关于未知目标标签

l_{T} (x_{1}^{'}), \dots, l_{T} (x_{t}^{'})

的信息，并且可以基于可用数据进行计算。方程 (11) 可以解释如下。

Lemma 1 (see the supplementary material for its proof) shows that the inequality in Eq. (12) is satisfied if

α_{j} \leq α_{i} \leq α^{*}

. This implies that

α_{i} > α^{*}

if the criterion is violated for some

α_{j} \leq α_{i}

. Consequently,the maximum

α_{BP}

as defined in Eq. (12) from an increasing sequence

α_{1}, \dots, α_{w}

which violates Eq. (13) for some

j \in {1, \dots, i - 1}

can be assumed to be near to

α^{*}

引理 1（其证明见补充材料）表明，如果

α_{j} \leq α_{i} \leq α^{*}

，则方程 (12) 中的不等式成立。这意味着如果某些

α_{j} \leq α_{i}

的标准被违反，则

α_{i} > α^{*}

。因此，从一个递增序列

α_{1}, \dots, α_{w}

中定义的最大值

α_{BP}

如果对某些

j \in {1, \dots, i - 1}

违反了方程 (13)，可以假设接近于

α^{*}

。

Theorem 1 shows that the target error of the model

g_{α_{BP}} \circ ϕ_{α_{BP}}

identified by the BPDA has the same error rate as

D (α^{*})

for

t \to \infty

. Moreover,if the optimum

inf_{α \in [0, \infty)} D (α) + \bar{E} (α)

is achieved,then the error rate is optimal in the sense that

ε_{T} (g_{α_{BP}} \circ ϕ_{α_{BP}})

is only a constant factor worse than the minimum

inf_{α \in [0, \infty)} D (α) + \bar{E} (α)

. The constant factor is given by

(3 + 3 B / D (0)) q max {D (0) / B, 1}

. That is,the bound is larger for steeper

D

between two consecutive values for

α

(

q

is larger),and,it is the supplementary material. If

E (α)

is increasing and the instantiation bound in Eq. (2) is tight,i.e. it holds with equality,then the error

ε_{T} (g_{α_{BP}} \circ ϕ_{α_{BP}})

is only a constant factor worse than the optimal error

inf_{α \in [0, \infty)} ε_{T} (g_{α} \circ ϕ_{α})

定理 1 显示，BPDA 确定的模型的目标误差

g_{α_{BP}} \circ ϕ_{α_{BP}}

与

D (α^{*})

的误差率相同，适用于

t \to \infty

。此外，如果达到最优

inf_{α \in [0, \infty)} D (α) + \bar{E} (α)

，则误差率在某种意义上是最优的，因为

ε_{T} (g_{α_{BP}} \circ ϕ_{α_{BP}})

仅比最小的

inf_{α \in [0, \infty)} D (α) + \bar{E} (α)

差一个常数因子。这个常数因子由

(3 + 3 B / D (0)) q max {D (0) / B, 1}

给出。也就是说，对于两个连续值之间的陡峭

D

，界限更大（

q

更大），并且这是补充材料。如果

E (α)

是递增的，并且公式 (2) 中的实例化界限是紧的，即它成立于等式，那么误差

ε_{T} (g_{α_{BP}} \circ ϕ_{α_{BP}})

仅比最优误差

inf_{α \in [0, \infty)} ε_{T} (g_{α} \circ ϕ_{α})

差一个常数因子。

It is well known that appropriate assumptions as given above are needed for successful domain adaptation

[52, 19]

. However,in practice,target error bounds are often not tight and optimality cannot be guaranteed. In the following section, we therefore investigate the performance of our method on benchmark datasets based on our two accompanying target error bounds.

众所周知，上述适当假设对于成功的领域适应

[52, 19]

是必要的。然而，在实践中，目标误差界限往往不紧，无法保证最优性。因此，在接下来的部分中，我们将基于我们的两个伴随目标误差界限，调查我们方法在基准数据集上的表现。

Datasets We rely on one academic example which we call Transformed Moons. Transformed Moons shows a density ratio that is unbounded in large regions, see Figure 1. We also use the Amazon Reviews dataset [53]. This dataset contains text reviews from four domains: books (B), DVDs (D), electronics (E), and kitchen appliances (K). Reviews are encoded in 5000 dimensional feature vectors of bag-of-words unigrams and bigrams with binary labels: label 0 if the product is ranked by 1 to 3 stars, and label 1 if the product is ranked by 4 or 5 stars. From the four categories we obtain twelve domain adaptation tasks where each category serves once as source domain and once as target domain. examples and 2000 unlabeled target examples for training, and between 3000 and 6000 examples for testing. Our third dataset is the DomainNet-2019 dataset consisting of six different domains, each having 345 classes, and an average count of 288 images per class, i.e. around 0.6 million images [56]. However, our focus is not on large-scale domain adaptation, but rather on ranking model selection methods; hence, we propose a reduced version of the DomainNet-2019 dataset. In particular, we reduce the number of classes to five. We call our new dataset MiniDomainNet. See the supplementary material for the dataset statistics.

数据集我们依赖于一个学术示例，称为变换月亮（Transformed Moons）。变换月亮在大区域内显示出无界的密度比，见图1。我们还使用亚马逊评论数据集 [53]。该数据集包含来自四个领域的文本评论：书籍（B）、DVD（D）、电子产品（E）和厨房电器（K）。评论被编码为5000维特征向量，使用词袋模型的单词和双词，带有二进制标签：如果产品的评级为1到3星，则标签为0；如果产品的评级为4或5星，则标签为1。我们从这四个类别中获得十二个领域适应任务，每个类别作为源领域和目标领域各服务一次。用于训练的示例和2000个未标记的目标示例，以及用于测试的3000到6000个示例。我们的第三个数据集是DomainNet-2019数据集，由六个不同的领域组成，每个领域有345个类别，每个类别平均有288张图像，即大约60万张图像 [56]。然而，我们的重点不在于大规模领域适应，而是在于排名模型选择方法；因此，我们提出了DomainNet-2019数据集的简化版本。特别是，我们将类别数量减少到五个。我们将新的数据集称为MiniDomainNet。有关数据集统计信息，请参见补充材料。

Balancing principle for domain adaptation We apply the BPDA in Algorithm 1 based on the two accompanying target error bounds described in Section 3. On the two-class datasets Transformed Moons and Amazon Reviews, we apply the BPDA based on Eq. (4). We set the domain distance

D (α) := {\hat{d}}_{G △ G} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{t, G, δ}

and approximate its value by a classifier as proposed in [16, Subsection 3.2]. In our experiments on the multi-class dataset MiniDomainNet, we apply the BPDA based on Eq. (6). We define

D (α) := {\hat{d}}_{h, H}^{(ρ)} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{s + t, H, δ}

and approximate its value as proposed in [20, Subsection 4.2]. In both variants of the BPDA, one for each bound, we follow the argument of [25, Subsection 7.2] to have enough unlabeled data to evaluate the bounds without considering the finite sample error term

η_{t, G, δ}

. We repeat each domain adaptation model training several times. The if-statement in the BPDA in Algorithm 1 is considered violated, if there is a violation of the statement for at least one of the repetitions. For a fair comparison, the evaluations of IWV and DEV are also based on all repetitions. More precisely, for IWV and DEV, we choose the parameter with the lowest average importance weighted risk and lowest average DEV-risk, respectively, where the average is computed over all repetitions.

域适应的平衡原则我们在算法 1 中应用 BPDA，基于第 3 节中描述的两个伴随目标误差界限。在两个类数据集 Transformed Moons 和 Amazon Reviews 上，我们基于公式 (4) 应用 BPDA。我们设定域距离

D (α) := {\hat{d}}_{G △ G} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{t, G, δ}

并通过分类器近似其值，如 [16, 子节 3.2] 中所提议。在我们对多类数据集 MiniDomainNet 的实验中，我们基于公式 (6) 应用 BPDA。我们定义

D (α) := {\hat{d}}_{h, H}^{(ρ)} (ϕ_{α} (x), ϕ_{α} (x^{'})) + η_{s + t, H, δ}

并根据 [20, 子节 4.2] 中的提议近似其值。在 BPDA 的两个变体中，每个界限一个，我们遵循 [25, 子节 7.2] 的论点，以确保有足够的未标记数据来评估界限，而不考虑有限样本误差项

η_{t, G, δ}

。我们对每个域适应模型的训练进行了多次重复。如果在 BPDA 的算法 1 中的 if 语句对于至少一次重复被违反，则认为该语句被违反。为了公平比较，IWV 和 DEV 的评估也基于所有重复。更准确地说，对于 IWV 和 DEV，我们选择具有最低平均重要性加权风险和最低平均 DEV 风险的参数，其中平均值是基于所有重复计算的。

Parameter choice baselines We compare our parameter choice approach to four baselines. The first baseline is training on source data only (SO). The second baseline is the best target error (TB) and it serves as a lower bound for the error. The third baseline is importance weighted validation [3] (IWV). We follow [23] and use held-out validation, i.e. we hold out a part of the training data as validation set, and we compute the importance weights based on this validation set. We also follow [57] and [23, Subsection 4.3] to estimate the importance weight by a classifier trained to separate source from target data. The classifier is tuned separately for each task and dataset such that its validation misclassification error is at most 0.05 . For MiniDomainNet, we compute the importance weight based on the features of the pre-trained ResNet-18 [58]. The fourth baseline is deep embedded validation [23] (DEV) which solves the unbounded variance issue in IWV. Following [23], DEV is applied on the features of the neural networks.

参数选择基准我们将我们的参数选择方法与四个基准进行比较。第一个基准仅在源数据上进行训练（SO）。第二个基准是最佳目标误差（TB），它作为误差的下限。第三个基准是重要性加权验证 [3]（IWV）。我们遵循 [23] 的方法，使用保留验证，即我们将一部分训练数据保留作为验证集，并基于该验证集计算重要性权重。我们还遵循 [57] 和 [23, 子节 4.3] 的方法，通过一个训练来区分源数据和目标数据的分类器来估计重要性权重。该分类器为每个任务和数据集单独调整，以确保其验证错误率最多为 0.05。对于 MiniDomainNet，我们基于预训练的 ResNet-18 [58] 的特征计算重要性权重。第四个基准是深度嵌入验证 [23]（DEV），它解决了 IWV 中的无界方差问题。遵循 [23]，DEV 应用于神经网络的特征上。

Domain adaptation methods In our experiments, we use three domain adaptation techniques. Domain-adversarial neural networks [16] (DANN), Maximum Mean Discrepancy [59, 15] (MMD) and Central Moment Discrepancy [18] (CMD). The details of all neural network architectures used, as well as the training strategy and hyperparameters are provided in the supplementary material.

领域适应方法在我们的实验中，我们使用了三种领域适应技术。领域对抗神经网络 [16]（DANN）、最大均值差异 [59, 15]（MMD）和中心矩差异 [18]（CMD）。所有使用的神经网络架构的详细信息，以及训练策略和超参数，均在补充材料中提供。

Assumptions To evaluate the compliance of the assumptions in Subsection 4.1 for the algorithms CMD and MMD,we estimate

E

,the domain distance

D

and the least non-decreasing majorant

\bar{E}

on the Transformed Moons dataset. In particular,the term

E

is computed using the (in unsupervised combined error in Eq. (4). We repeat the evaluations 10 times for different initializations of the domain adaptation model weights.

假设为了评估算法 CMD 和 MMD 在子节 4.1 中假设的合规性，我们在转化的 Moons 数据集上估计

E

、领域距离

D

和最小非递减上界

\bar{E}

。特别地，项

E

是使用（在无监督组合误差中）计算的，见公式 (4)。我们对领域适应模型权重的不同初始化重复评估 10 次。

Assumptions The terms

E

and

D

for MMD on Transformed Moons are shown in Figure 2 and, for CMD, in the supplementary material. The following observation (a)-(d) can be made: (a)

E

is bounded,the mean curves of

D (α)

tend to be non-increasing and the technical assumption

D (α) / D (0) < sup_{α \in [0, \infty)} E (α) / B

is satisfied for

α \geq 10^{3}

. That is,the mean curves tend to follow the assumptions listed in Section 4.1. (b) The mean curves

\bar{E}

and

E

tend to be similar. (c) The average parameters chosen by the balancing principle (see

α_{BP}

in Figure 2) are the maximum values for which the mean curves of

\bar{E} (α) / B

are smaller than the mean curves of

D (α) / D (0)

. That is,the BPDA described in Algorithm 1 tends to perform as expected. (d) There is a moderate trend towards a figure). However, the BPDA is (on average) robust w.r.t. this violation as it picks nearly the optimal value (see Table 2).

假设 MMD 在转化的 Moons 上的项

E

和

D

如图 2 所示，而 CMD 的项则在补充材料中展示。可以做出以下观察 (a)-(d)：(a)

E

是有界的，

D (α)

的均值曲线趋向于非递减，并且技术假设

D (α) / D (0) < sup_{α \in [0, \infty)} E (α) / B

对

α \geq 10^{3}

是满足的。也就是说，均值曲线趋向于遵循第 4.1 节中列出的假设。(b) 均值曲线

\bar{E}

和

E

趋向于相似。(c) 平衡原则选择的平均参数（见图 2 中的

α_{BP}

）是均值曲线

\bar{E} (α) / B

小于均值曲线

D (α) / D (0)

的最大值。也就是说，算法 1 中描述的 BPDA 趋向于按预期执行。(d) 存在向某个图形的适度趋势。然而，BPDA 在此违反方面（平均而言）是稳健的，因为它选择了几乎最优的值（见表 2）。

Comparative study Average values of our empirical evaluations on Transformed Moons and Amazon Review are summarized in Table 2. For the full tables on Transformed Moons, Amazon Reviews and MiniDomainNet, see the supplementary material. Although making no assumptions on the ratio between target and source density, the BPDA outperforms related parameter choice methods (IWV, DEV) on almost all tasks on Transformed Moons and Amazon Reviews, and, obtains competitive results on MiniDomainNet.

比较研究我们在变换月球和亚马逊评论上的实证评估的平均值总结在表2中。有关变换月球、亚马逊评论和MiniDomainNet的完整表格，请参见补充材料。尽管对目标和源密度之间的比率没有假设，BPDA在变换月球和亚马逊评论的几乎所有任务上都优于相关的参数选择方法（IWV，DEV），并在MiniDomainNet上获得了具有竞争力的结果。

One widely-used technique for unsupervised domain adaptation is to map the data into a new feature space where the source and target data representations appear similar, and where enough information is preserved for prediction [14]. The similarity is often realized by minimizing the source error weighted by a distance measure between source and target representations. One common interpretation of this approach is to learn domain-invariant representations. However, there is a subtle difference between distance-regularization by penalizing the error minimization with a distance as above, and distance-minimization which results in domain-invariant representations. The latter can even lead to deteriorated performance [19]. In contrast, the interpretation as regularization problem opens up a powerful toolbox of mathematical techniques. We take up the technique of balancing stability and approximation in the regularization of ill-posed inverse problems, to tackle the problem of choosing the crucial regularization parameter in distance-regularized domain adaptation. Our approach overcomes the lack of target labels, it satisfies a generalization guarantee, and is (to the best of our knowledge) the first theoretically justified method that allows source and target distributions with disjoint supports. Finally, our approach outperforms or is on par, with the state of the art on the problem of choosing the regularization parameter, on several domain adaptation methods; applied on different datasets.

一种广泛使用的无监督领域适应技术是将数据映射到一个新的特征空间，在该空间中，源数据和目标数据的表示看起来相似，并且保留了足够的信息以进行预测 [14]。相似性通常通过最小化源误差并加权源和目标表示之间的距离度量来实现。这种方法的一个常见解释是学习领域不变的表示。然而，通过用距离惩罚误差最小化的距离正则化与导致领域不变表示的距离最小化之间存在微妙的区别。后者甚至可能导致性能下降 [19]。相比之下，将其解释为正则化问题打开了一套强大的数学技术工具箱。我们采用在不适定逆问题的正则化中平衡稳定性和近似的技术，以解决在距离正则化领域适应中选择关键正则化参数的问题。我们的方法克服了目标标签的缺乏，满足了一般化保证，并且（据我们所知）是首个理论上有依据的方法，允许源和目标分布具有不相交的支持。最后，我们的方法在选择正则化参数的问题上，在几种领域适应方法中超越或与最先进的技术相当；并应用于不同的数据集。

Many fields, such as manufacturing, personalized medicine or analytical chemistry, have to handle problems of domain shift together with issues of data limitations. These areas can profit from our research, as it provides a principled way of choosing a justified regularization parameter of unsupervised domain adaptation algorithms. Our method provides a high level of trust by applying mathematical techniques with guarantees originally developed in the area of ill-posed inverse problems. However, in domain adaptation in general, one critical point is the bias stored in the source data (e.g. past product, previous patient data, previous spectroscope), since knowledge in this data is used to improve the learning on the unlabeled target data (e.g. new product, new patient data, new spectroscope). If such a bias is present in one of the source domains, the predictions in new target domains might always suffer from this bias. We therefore suggest to not only look at new data in new target domains but more importantly consider dataset bias in already collected labeled source data.

许多领域，如制造业、个性化医学或分析化学，必须处理领域转移问题以及数据限制问题。这些领域可以从我们的研究中受益，因为它提供了一种选择无监督领域适应算法中合理正则化参数的原则性方法。我们的方法通过应用最初在病态逆问题领域开发的具有保证的数学技术，提供了高度的信任。然而，在领域适应中，一个关键点是存储在源数据中的偏差（例如，过去的产品、以前的患者数据、以前的光谱仪），因为这些数据中的知识用于改善对未标记目标数据（例如，新产品、新患者数据、新光谱仪）的学习。如果这种偏差存在于某个源领域中，那么在新的目标领域中的预测可能总是受到这种偏差的影响。因此，我们建议不仅要关注新目标领域中的新数据，更重要的是考虑已经收集的标记源数据中的数据集偏差。

We thank Markus Holzleitner for careful proofreading. We thank the anonymous reviewers for a constructive discussion which formed the basis for the supplementary material's section about limitations, risks and future developments. The research reported in this paper has been funded by the Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK), the Federal Ministry for Digital and Economic Affairs (BMDW), and the Province of Upper Austria in the frame of the COMET-Competence Centers for Excellent Technologies Programme and the COMET Module S3AI managed by the Austrian Research Promotion Agency FFG. The LIT AI Lab is financed by the Federal State of Upper Austria. We further acknowledge the partial support of the FFG project AutoQual-I. The ELLIS Unit Linz, the LIT AI Lab and the Institute for Machine Learning, are the projects AI-MOTION (LIT-2018-6-YOU-212), DeepToxGen (LIT-2017-3-YOU-003), AI-SNN (LIT-2018-6-YOU-214), DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), DL for GranularFlow (FFG- 871302), AIRI FG 9-N (FWF-36284, FWF-36235), ELISE (H2020-ICT-2019-3 ID: 951847), AIDD (MSCA-ITN-2020 ID: 956832). We thank Janssen Pharmaceutica (MaDeSMart, HBC.2018.2287), Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, TÜV Austria, and the NVIDIA program with the award GCP19980904.

我们感谢Markus Holzleitner的仔细校对。我们感谢匿名评审者的建设性讨论，这为关于局限性、风险和未来发展的补充材料部分奠定了基础。本文所报告的研究得到了联邦气候行动、环境、能源、交通、创新和技术部（BMK）、联邦数字和经济事务部（BMDW）以及上奥地利省在COMET卓越技术中心项目框架下的资助，以及由奥地利研究促进机构FFG管理的COMET模块S3AI。LIT AI实验室由上奥地利联邦州资助。我们还感谢FFG项目AutoQual-I的部分支持。ELLIS单位林茨、LIT AI实验室和机器学习研究所的项目包括AI-MOTION（LIT-2018-6-YOU-212）、DeepToxGen（LIT-2017-3-YOU-003）、AI-SNN（LIT-2018-6-YOU-214）、DeepFlood（LIT-2019-8-YOU-213）、医学认知计算中心（MC3）、INCONTROL-RL（FFG-881064）、PRIMAL（FFG-873979）、DL for GranularFlow（FFG-871302）、AIRI FG 9-N（FWF-36284，FWF-36235）、ELISE（H2020-ICT-2019-3 ID: 951847）、AIDD（MSCA-ITN-2020 ID: 956832）。我们感谢Janssen Pharmaceutica（MaDeSMart，HBC.2018.2287）、Audi.JKU深度学习中心、TGW LOGISTICS GROUP GMBH、Silicon Austria Labs（SAL）、FILL Gesellschaft mbH、Anyline GmbH、谷歌、ZF Friedrichshafen AG、罗伯特·博世有限公司、UCB Biopharma SRL、默克医疗保健KGaA、Verbund AG、奥地利TÜV以及获得奖项GCP19980904的NVIDIA项目。

The parameter selection methods for domain adaptation require to train several models with various parameters. Furthermore, evaluation of such methods include applying various domain adaptation techniques, which results in high computational demand when using large-scale datasets. In order to reduce the computational resources needed in this area, while keeping the difficulty of working with high-resolution images, and working on a problem with several domains, we fork a smaller version of the DomainNet dataset [4], which we call the MiniDomainNet dataset. MiniDomainNet makes research on the area of parameter selection for domain adaptation more accessible, by significantly reducing the computational needs, while providing a challenging, and sufficiently-large test bed for evaluating deep models.

域适应的参数选择方法需要训练多个具有不同参数的模型。此外，这些方法的评估包括应用各种域适应技术，这在使用大规模数据集时会导致高计算需求。为了减少在这一领域所需的计算资源，同时保持处理高分辨率图像的难度，并处理多个领域的问题，我们从 DomainNet 数据集中派生出一个较小的版本，称为 MiniDomainNet 数据集。MiniDomainNet 通过显著降低计算需求，使得在域适应参数选择领域的研究更加可及，同时提供了一个具有挑战性且足够大的测试平台，用于评估深度模型。

The DomainNet dataset consists of approximately 0.6 million images divided into 6 domains (Quick-draw, Real, Clipart, Sketch, Infograph, and Painting), with each domain having 345 classes. The average count of images of DomainNet in each class, and across all domains is approx. 288. We curate the MiniDomainNet dataset from the DomainNet dataset as follows. We select the top-five largest classes in the training set of DomainNet, based on the highest average image-count per class across all domains. This selection process will result in a dataset with the largest amount of training data per class, which is ideal for training deep models.

DomainNet 数据集由大约 60 万张图像组成，分为 6 个领域（快速绘制、真实、剪贴画、素描、信息图和绘画），每个领域有 345 个类别。DomainNet 中每个类别的平均图像数量在所有领域中约为 288。我们从 DomainNet 数据集中策划 MiniDomainNet 数据集，具体如下。我们根据所有领域中每个类别的平均图像数量选择训练集中前五个最大的类别。这个选择过程将导致每个类别拥有最大数量的训练数据，这对于训练深度模型是理想的。

In our experiments with MiniDomainNet, we follow a recommendation in [4], that uses a combined-source setting. To define our domain adaptation tasks, we select 5 out of the 6 domains and combine them into our combined source dataset. We use the remaining domain as our target dataset. By permuting all source combinations, we then define 6 domain adaptation tasks, which we refer to as combined-source datasets (CS, as denoted in Table 5).

在我们对 MiniDomainNet 的实验中，我们遵循了文献 [4] 中的建议，采用了组合源设置。为了定义我们的领域适应任务，我们从 6 个领域中选择 5 个，并将它们组合成我们的组合源数据集。我们使用剩余的领域作为我们的目标数据集。通过排列所有源组合，我们定义了 6 个领域适应任务，我们称之为组合源数据集（CS，如表 5 所示）。

In addition to providing the MiniDomainNet, we further address an issue regarding the currently available version of the DomainNet dataset. During our development process, we found that 10 files from the class 327 (t-shirt) in the painting domain sub-set, were missing in the file list of the training set (painting_train.txt). We provide a fix for this issue in our source code, by inserting the missing class references and their corresponding files. The fix can be found in the source code, in dataloaders/domainnet.py.

除了提供 MiniDomainNet，我们还进一步解决了当前可用版本的 DomainNet 数据集的问题。在我们的开发过程中，我们发现绘画领域子集中类别 327（T恤）的 10 个文件在训练集的文件列表中缺失（painting_train.txt）。我们在源代码中提供了对此问题的修复，通过插入缺失的类别引用及其对应的文件。修复可以在源代码的 dataloaders/domainnet.py 中找到。

Transformed Moons On the Transformed Moons dataset, we use a feed-forward network with two fully-connected layers, with 16 nodes each, followed by ReLU non-linearity. The network is optimized by Adam [5] optimizer for 250 epochs,with

β_{1} = 0.9, β_{2} = 0.999

,and the initial learning rate of 0.01 , using a MultiStep scheduler which halved the learning rate in epochs 50,100 , and 150 . To train proxy-A classifiers (required in [6]), we use 1 fully-connected layer with 16 nodes; and are trained with Adam optimizer for 200 epochs,with

β_{1} = 0.9, β_{2} = 0.999

,and an initial learning rate of 0.01,and a MultiStep scheduler halving learning rate on epochs 50,100,150 .

转换后的月球在转换后的月球数据集上，我们使用一个前馈网络，该网络具有两个完全连接的层，每层16个节点，后接ReLU非线性。该网络通过Adam [5] 优化器优化250个周期，使用

β_{1} = 0.9, β_{2} = 0.999

，初始学习率为0.01，采用MultiStep调度器，在第50、100和150个周期将学习率减半。为了训练代理-A分类器（在[6]中需要），我们使用1个完全连接的层，包含16个节点；并使用Adam优化器训练200个周期，使用

β_{1} = 0.9, β_{2} = 0.999

，初始学习率为0.01，并在第50、100、150个周期将学习率减半的MultiStep调度器。

Amazon Reviews For Amazon Reviews, we follow [7] and use a feed-forward network three fully-connected layers, with 100 nodes each, and sigmoid non-linearity. The optimizer, learning rate, and scheduler are the same as in the Transformed Moons experiments (see above). We train each model for CMD and MMD experiments for 50 epochs and for DANN for 500 . To estimate the

G △ G

-divergence,we follow [7] and train a classifier for separating the source sample and the target sample. In particular we apply 2 fully-connected layers with 100 nodes each and use the Adam optimizer for 200 epochs,with

β_{1} = 0.9, β_{2} = 0.999

,the initial learning rate of 0.01,and,a MultiStep scheduler halving learning rate on epochs 50,100,150.

亚马逊评论对于亚马逊评论，我们遵循[7]的做法，使用一个前馈网络，该网络具有三个完全连接的层，每层100个节点，并使用sigmoid非线性。优化器、学习率和调度器与转换后的月球实验相同（见上文）。我们为CMD和MMD实验训练每个模型50个周期，为DANN训练500个周期。为了估计

G △ G

-散度，我们遵循[7]，训练一个分类器以区分源样本和目标样本。具体而言，我们应用2个完全连接的层，每层100个节点，并使用Adam优化器训练200个周期，使用

β_{1} = 0.9, β_{2} = 0.999

，初始学习率为0.01，并在第50、100、150个周期将学习率减半的MultiStep调度器。

MiniDomainNet Following the pre-trained setup from [4], we use a frozen ResNet-18 backbone model which was trained on ImageNet [8], and operate subsequent computations on the 512 dimensional extracted features. To alleviate overfitting effects on pre-computed features, we perform data augmentation on each batch and forward the images through the backbone each time. We incorporate zero padding before resizing the images to

256 \times 256

to avoid image distortions. Following the guidance for data augmentation techniques from [9],we perform random resized cropping to

224 \times 224

with a random viewport between

70 %

and

100 %

of the original image,random horizontal flipping, color jittering of

0.25 %

on each RGB channel,and a

\pm 2

degree rotation. After the ResNet-18 backbone output, we add several projection layers, and define the domain adaptation layers on which we use the domain adaptation methods to align the representations. The first layers are defined as a common architecture across the different domain adaptation methods. Additional layers are further added for the classification networks, according to the requirements of the individual domain adaptation methods in CMD or MMD. The number of layers/neurons in the upper layers of our architecture have been tuned in order to achieve the best performance in the source-only setup. See Table 1 for a detailed description of the architecture used. We perform experiments on all 6 domain adaptation tasks as defined in 5.4 for each of the previously listed methods. All methods have been trained for 50 epochs with Adam optimizer,an initial learning rate of

0.001, β_{1} = 0.9, β_{2} = 0.999

, and a MultiStep learning rate scheduler, halving the learning rate after 15 and 35 epochs.

MiniDomainNet 根据 [4] 的预训练设置，我们使用一个冻结的 ResNet-18 主干模型，该模型在 ImageNet [8] 上进行了训练，并对提取的 512 维特征进行后续计算。为了减轻对预计算特征的过拟合影响，我们对每个批次进行数据增强，并在每次通过主干时转发图像。在调整图像大小到

256 \times 256

之前，我们加入零填充，以避免图像失真。根据 [9] 的数据增强技术指导，我们对原始图像进行随机缩放裁剪到

224 \times 224

，随机视口在

70 %

和

100 %

之间，随机水平翻转，对每个 RGB 通道进行

0.25 %

的颜色抖动，以及进行

\pm 2

度的旋转。在 ResNet-18 主干输出后，我们添加了几个投影层，并定义了领域适应层，在这些层上我们使用领域适应方法来对齐表示。第一层被定义为不同领域适应方法之间的通用架构。根据 CMD 或 MMD 中各个领域适应方法的要求，进一步添加了额外的层用于分类网络。我们在架构的上层调整了层数/神经元的数量，以实现源仅设置下的最佳性能。有关所使用架构的详细描述，请参见表 1。我们对在 5.4 中定义的所有 6 个领域适应任务进行了实验，针对之前列出的每种方法。所有方法均使用 Adam 优化器训练 50 个周期，初始学习率为

0.001, β_{1} = 0.9, β_{2} = 0.999

，并使用 MultiStep 学习率调度器，在第 15 和 35 个周期后将学习率减半。

To apply the balancing principle, we require the training of an additional MDD classifier, see [10]), using the features of the adaptation layer from CMD and MMD, which is further used to calculate the MDD distance. The architecture of the MDD classifier is listed in Table 2. The MDD classifiers are trained with Adam optimizer, initial learning rate of 0.0001 , and a MultiStep scheduler halving the learning rate after 15 and 25 epochs, and in total we run them for 35 epochs. For selecting the disparity parameters we followed the guidance from [11],and set

γ = 1.1

in the MDD training loss, and

ρ = 0

for calculating the MDD distance employed in BPDA.

为了应用平衡原则，我们需要训练一个额外的 MDD 分类器，参见 [10]，使用来自 CMD 和 MMD 的适应层特征，这进一步用于计算 MDD 距离。MDD 分类器的架构列在表 2 中。MDD 分类器使用 Adam 优化器进行训练，初始学习率为 0.0001，并使用 MultiStep 调度器在 15 和 25 个周期后将学习率减半，总共运行 35 个周期。为了选择差异参数，我们遵循了 [11] 的指导，并在 MDD 训练损失中设置了

γ = 1.1

，在 BPDA 中计算 MDD 距离时设置了

ρ = 0

。

In experiments on Transformed Moons and Amazon Reviews, we used two HPC stations with in total 8xNVIDIA TITAN RTX 24GB, 4xIntel Xeon Scalable Processors Skylake Gold 6130 (2.10 GHz) and Ubuntu 18.04. All methods have been implemented in python using the Pytorch library [12]. We use Scikit-learn library [13] for evaluation measures and toy datasets, and the TQDM library [14], and Tensorboard [15] for keeping track of the progress of our experiments.

在对变换月亮和亚马逊评论的实验中，我们使用了两个 HPC 站，总共配备了 8 个 NVIDIA TITAN RTX 24GB、4 个 Intel Xeon 可扩展处理器 Skylake Gold 6130 (2.10 GHz) 和 Ubuntu 18.04。所有方法均在 Python 中实现，使用 Pytorch 库 [12]。我们使用 Scikit-learn 库 [13] 进行评估指标和玩具数据集，并使用 TQDM 库 [14] 和 Tensorboard [15] 来跟踪实验进展。

Transformed Moons and Amazon Reviews IWV [16], DEV [11], and BPDA are used to choose the best parameter

α \in {0, 10^{- 6}, 10^{- 5}, \dots, 10^{3}, 10^{4}}

,for three different distance-regularized domain adaptation methods, namely DANN [7], MMD [17, 18] and CMD [19]. The Transformed Moons and Amazon Reviews datasets contain only 2 classes; hence, BPDA is employed using the

变换月亮和亚马逊评论 IWV [16]、DEV [11] 和 BPDA 被用来选择最佳参数

α \in {0, 10^{- 6}, 10^{- 5}, \dots, 10^{3}, 10^{4}}

，用于三种不同的距离正则化领域适应方法，即 DANN [7]、MMD [17, 18] 和 CMD [19]。变换月亮和亚马逊评论数据集仅包含 2 个类别；因此，使用 BPDA。

Figure 2: Average and standard deviation over 10 repetitions of estimated learning errors

E

(in unsupervised domain adaptation unknown) and the domain distance

D

of the accompanying target error bound [6] for distance-regularized domain adaptation models with the Maximum Mean Discrepancy [20] (left) and the Central Moment Discrepancy (right).

图 2: 在 10 次重复中估计的学习错误的平均值和标准差

E

（在无监督领域适应中未知）以及伴随的目标错误界限 [6] 的领域距离

D

，适用于具有最大均值差异 [20]（左）和中心矩差异（右）的距离正则化领域适应模型。

This section provides empirical evidence for the compliance of the empirical settings with our assumptions made in Section 4.1 of the main document, and, gives results on the three aforementioned datasets, comparing our approach with the state of the art in parameter selection for domain adaptation.

本节提供了实证证据，证明实证设置与我们在主文档第 4.1 节中所做假设的一致性，并给出了在三个上述数据集上的结果，将我们的方法与领域适应参数选择的最新技术进行了比较。

Table 3: Average target classification error (and standard deviation) for different regularization parameter choices on the Transformed Moons dataset. 10 repetitions with different random initialization of model weights are used to estimate the importance weighted risk, the DEV risk and the BPDA. The BPDA is computed using the bound in [6].

表 3：在变换月亮数据集上，不同正则化参数选择的平均目标分类错误（及标准差）。使用 10 次不同随机初始化模型权重的重复来估计重要性加权风险、DEV 风险和 BPDA。BPDA 是使用 [6] 中的界限计算的。

Method	SO	IWV	DEV	BPDA	TB
MMD	$0.205 (\pm 0.025)$	0.199 (±0.031)	$0.339 (\pm 0.065)$	$0.157 (\pm 0.069)$	$0.157 (\pm 0.069)$
DANN	0.177 (±0.032)	$0.177 (\pm 0.032)$	0.169 (±0.075)	0.115 (±0.098)	$0.115 (\pm 0.098)$
CMD	$0.205 (\pm 0.026)$	0.198 (±0.022)	$0.190 (\pm 0.051)$	0.185 (±0.039)	$0.181 (\pm 0.038)$
Avg.	$0.196 (\pm 0.028)$	0.191 (±0.028)	$0.232 (\pm 0.064)$	0.152 (±0.069)	$0.151 (\pm 0.068)$

方法	SO	IWV	DEV	BPDA	TB
MMD	$0.205 (\pm 0.025)$	0.199 (±0.031)	$0.339 (\pm 0.065)$	$0.157 (\pm 0.069)$	$0.157 (\pm 0.069)$
DANN	0.177 (±0.032)	$0.177 (\pm 0.032)$	0.169 (±0.075)	0.115 (±0.098)	$0.115 (\pm 0.098)$
CMD	$0.205 (\pm 0.026)$	0.198 (±0.022)	$0.190 (\pm 0.051)$	0.185 (±0.039)	$0.181 (\pm 0.038)$
平均值	$0.196 (\pm 0.028)$	0.191 (±0.028)	$0.232 (\pm 0.064)$	0.152 (±0.069)	$0.151 (\pm 0.068)$

for three different distance-regularized domain adaptation methods, namely DANN, MMD, and CMD. The 4 domains contained in Amazon Reviews are denoted in the tables as: books (B), DVDs (D), classification error across all tasks using the MMD method. These results are consistent across all domain adaptation techniques.

针对三种不同的距离正则化领域适应方法，即 DANN、MMD 和 CMD。亚马逊评论中的 4 个领域在表中表示为：书籍 (B)、DVD (D)、使用 MMD 方法在所有任务上的分类错误。这些结果在所有领域适应技术中是一致的。

MiniDomainNet Table 5 shows the results of the three model selection methods IWV, DEV, and BPDA. We omitted the experiments with DANN on the MiniDomainNet dataset due to our computational limits. The 6 domains in the MiniDomainNet are denoted in the tables as: Quickdraw (Q), Real (R), Sketch (S), Clipart (C), Infograph (I), and Painting (P). Since the source domain is always a combination of all the other domains except the target, we refer to the source as Combined Source (CS). As can be seen, our method achieves the lowest averaged classification error across all tasks using the CMD method. When using MMD, the three parameter selection methods perform very similar on average, with BPDA and DEV achieving the lowest average error across all tasks.

MiniDomainNet 表 5 显示了三种模型选择方法 IWV、DEV 和 BPDA 的结果。由于计算限制，我们省略了在 MiniDomainNet 数据集上使用 DANN 的实验。MiniDomainNet 中的 6 个领域在表中表示为：快速绘图 (Q)、真实 (R)、草图 (S)、剪贴画 (C)、信息图 (I) 和绘画 (P)。由于源领域始终是除目标领域之外所有其他领域的组合，我们将源领域称为组合源 (CS)。可以看出，我们的方法在使用 CMD 方法时在所有任务上实现了最低的平均分类错误。当使用 MMD 时，三种参数选择方法的平均表现非常相似，其中 BPDA 和 DEV 在所有任务上实现了最低的平均错误。

Label shift In this work,we do not assume a unique labeling function

l_{S} = l_{T}

for source and target domain (covariate shift assumption),but

l_{S}

and

l_{T}

should be similar. In fact,even if the labeling functions are different

l_{S} \neq l_{T}

,the balancing value

α^{*}

can be well estimated by the balancing principle estimate

α_{BP}

as proven by Lemma 1. However,the quality of the balancing value

α^{*}

itself can be negatively affected by excessive label shift. For example,the function

E

can first increase strongly and then decrease (caused by label shift) which implies an increasing difference between

\bar{E} (α)

and

E (α)

for increasing

α

. In such situations,the target error of the model

g_{α^{*}} \circ ϕ_{α^{*}}

can be high and consequently also the one of the model

g_{α_{BP}} \circ ϕ_{α_{BP}}

identified by the BPDA. However,note that our experiments (see also Figure 2) indicate similar values for

E

and

\bar{E}

标签转移在本研究中，我们不假设源域和目标域有唯一的标记函数

l_{S} = l_{T}

（协变量转移假设），但

l_{S}

和

l_{T}

应该是相似的。事实上，即使标记函数不同

l_{S} \neq l_{T}

，平衡值

α^{*}

也可以通过平衡原理估计

α_{BP}

来很好地估计，正如引理 1 所证明的。然而，平衡值

α^{*}

本身的质量可能会受到过度标签转移的负面影响。例如，函数

E

可以先强烈增加然后减少（由标签转移引起），这意味着随着

α

的增加，

\bar{E} (α)

和

E (α)

之间的差异也在增加。在这种情况下，模型

g_{α^{*}} \circ ϕ_{α^{*}}

的目标误差可能很高，因此通过 BPDA 确定的模型

g_{α_{BP}} \circ ϕ_{α_{BP}}

的目标误差也会很高。然而，请注意我们的实验（见图 2）表明

E

和

\bar{E}

的值相似。

Loose instantiation bound Our model is agnostic w.r.t. the property that different target error bounds can be chosen as a basis. However,it is known in inverse problem literature

[21, 3]

that a loose bound can lead to a low performance of the balancing principle which also holds for the BPDA. This problem can be approached by choosing target error bounds that take into account the specific domain shift situation, e.g. the bound [22] is suitable for general domain shift scenarios.

松散实例化界限我们的模型对不同目标误差界限可以作为基础的属性是不可知的。然而，在逆问题文献中已知

[21, 3]

，松散界限可能导致平衡原理的低性能，这对于 BPDA 也是适用的。这个问题可以通过选择考虑特定领域转移情况的目标误差界限来解决，例如界限 [22] 适用于一般领域转移场景。

Low performance of all models Situations exist which hinder distance-regularized domain adaptation methods to perform well. One such situation is excessive label shift as discussed above, see also [23] and references therein. Such scenarios can cause all models

f_{1}, \dots, f_{w}

to be inaccurate or unstable. The BPDA will select a model

f_{i} \in {f_{1}, \dots, f_{n}}

with low target error

ε_{T} (f_{i})

compared to other models in the set

{f_{1}, \dots, f_{n}}

. Nevertheless,in such situations,

f_{i}

might have a high target error

ε_{T} (f_{i})

or it might be unstable. As a solution,distance-regularized domain adaptation methods can be applied with modifications, see e.g. [22, 24, 25, 26].

所有模型的低性能存在一些情况妨碍距离正则化领域适应方法的良好表现。上述讨论的过度标签偏移就是其中之一，参见 [23] 及其引用文献。这些场景可能导致所有模型

f_{1}, \dots, f_{w}

不准确或不稳定。BPDA 将选择一个目标误差

ε_{T} (f_{i})

较低的模型

f_{i} \in {f_{1}, \dots, f_{n}}

，与集合中的其他模型

{f_{1}, \dots, f_{n}}

相比。然而，在这种情况下，

f_{i}

可能具有较高的目标误差

ε_{T} (f_{i})

或可能不稳定。作为解决方案，可以对距离正则化领域适应方法进行修改，参见例如 [22, 24, 25, 26]。

Focus on weight parameter Our theoretical guarantees and the high empirical performance come at the cost of focusing the selection process on the distance-penalizing parameter

α

. This is in contrast to other model selection methods, such as [16, 11, 27, 28, 29], which can select different types of parameters. This limitation can be approached by employing ideas from multipenalty regularization of inverse problems to combine the BPDA with related approaches [30]. One approach is to explore a grid of admissible values of several parameters by applying the balancing principle with respect to one of them and allowing others to take all corresponding grid values. As the result of such an application, one obtains a much reduced set of parameter combinations of interest and one can apply the balancing principle or related approaches w.r.t. the other parameters. See e.g. [31] for a recent application of this approach.

关注权重参数我们的理论保证和高实证性能是以将选择过程集中在距离惩罚参数

α

上为代价的。这与其他模型选择方法形成对比，例如 [16, 11, 27, 28, 29]，这些方法可以选择不同类型的参数。可以通过借鉴逆问题的多重惩罚正则化的思想，将 BPDA 与相关方法 [30] 结合来解决这一限制。一种方法是通过应用平衡原则来探索多个参数的可接受值的网格，并允许其他参数取所有相应的网格值。通过这种应用，可以获得一个大大减少的感兴趣参数组合集，并可以针对其他参数应用平衡原则或相关方法。参见例如 [31] 以获取该方法的最新应用。

Task	MMD
Task	SO	IWV	DEV	BPDA	TB
$B \to D$	$0.225 (\pm 0.004)$	0.190 (±0.004)	$0.211 (\pm 0.005)$	$0.190 (\pm 0.004)$	$0.190 (\pm 0.004)$
$B \to E$	0.307 (±0.010)	$0.307 (\pm 0.010)$	$0.211 (\pm 0.005)$	$0.221 (\pm 0.008)$	$0.206 (\pm 0.012)$
$B \to K$	$0.266 (\pm 0.004)$	$0.185 (\pm 0.009)$	$0.266 (\pm 0.004)$	$0.185 (\pm 0.009)$	$0.185 (\pm 0.009)$
$D \to B$	$0.278 (\pm 0.008)$	$0.240 (\pm 0.007)$	$0.268 (\pm 0.006)$	$0.243 (\pm 0.006)$	$0.230 (\pm 0.007)$
$D \to E$	$0.273 (\pm 0.004)$	$0.273 (\pm 0.007)$	$0.249 (\pm 0.007)$	$0.207 (\pm 0.004)$	$0.189 (\pm 0.008)$
$D \to K$	$0.266 (\pm 0.004)$	$0.266 (\pm 0.004)$	0.197 (±0.006)	$0.197 (\pm 0.006)$	$0.187 (\pm 0.007)$
$E \to B$	$0.306 (\pm 0.003)$	$0.306 (\pm 0.003)$	$0.310 (\pm 0.007)$	$0.295 (\pm 0.013)$	$0.282 (\pm 0.014)$
$E \to D$	$0.307 (\pm 0.007)$	$0.285 (\pm 0.006)$	$0.288 (\pm 0.009)$	0.264 (±0.016	$0.255 (\pm 0.020)$
$E \to K$	$0.162 (\pm 0.004)$	0.145 (±0.003	0.193 (±0.004)	$0.145 (\pm 0.003)$	$0.145 (\pm 0.003)$
$K \to B$	$0.337 (\pm 0.007)$	$0.337 (\pm 0.007)$	$0.334 (\pm 0.006)$	$0.290 (\pm 0.010)$	$0.261 (\pm 0.010)$
$K \to D$	$0.293 (\pm 0.005)$	$0.294 (\pm 0.007)$	$0.306 (\pm 0.007)$	0.268 (±0.010)	$0.235 (\pm 0.014)$
$K \to E$	$0.167 (\pm 0.002)$	$0.169 (\pm 0.004)$	$0.167 (\pm 0.002)$	$0.169 (\pm 0.004)$	$0.145 (\pm 0.002)$
Avg.	$0.266 (\pm 0.005)$	$0.249 (\pm 0.005)$	$0.250 (\pm 0.005)$	$0.223 (\pm 0.008)$	$0.209 (\pm 0.009)$

任务	MMD
任务	SO	IWV	DEV	BPDA	TB
$B \to D$	$0.225 (\pm 0.004)$	0.190 (±0.004)	$0.211 (\pm 0.005)$	$0.190 (\pm 0.004)$	$0.190 (\pm 0.004)$
$B \to E$	0.307 (±0.010)	$0.307 (\pm 0.010)$	$0.211 (\pm 0.005)$	$0.221 (\pm 0.008)$	$0.206 (\pm 0.012)$
$B \to K$	$0.266 (\pm 0.004)$	$0.185 (\pm 0.009)$	$0.266 (\pm 0.004)$	$0.185 (\pm 0.009)$	$0.185 (\pm 0.009)$
$D \to B$	$0.278 (\pm 0.008)$	$0.240 (\pm 0.007)$	$0.268 (\pm 0.006)$	$0.243 (\pm 0.006)$	$0.230 (\pm 0.007)$
$D \to E$	$0.273 (\pm 0.004)$	$0.273 (\pm 0.007)$	$0.249 (\pm 0.007)$	$0.207 (\pm 0.004)$	$0.189 (\pm 0.008)$
$D \to K$	$0.266 (\pm 0.004)$	$0.266 (\pm 0.004)$	0.197 (±0.006)	$0.197 (\pm 0.006)$	$0.187 (\pm 0.007)$
$E \to B$	$0.306 (\pm 0.003)$	$0.306 (\pm 0.003)$	$0.310 (\pm 0.007)$	$0.295 (\pm 0.013)$	$0.282 (\pm 0.014)$
$E \to D$	$0.307 (\pm 0.007)$	$0.285 (\pm 0.006)$	$0.288 (\pm 0.009)$	0.264 (±0.016)	$0.255 (\pm 0.020)$
$E \to K$	$0.162 (\pm 0.004)$	0.145 (±0.003)	0.193 (±0.004)	$0.145 (\pm 0.003)$	$0.145 (\pm 0.003)$
$K \to B$	$0.337 (\pm 0.007)$	$0.337 (\pm 0.007)$	$0.334 (\pm 0.006)$	$0.290 (\pm 0.010)$	$0.261 (\pm 0.010)$
$K \to D$	$0.293 (\pm 0.005)$	$0.294 (\pm 0.007)$	$0.306 (\pm 0.007)$	0.268 (±0.010)	$0.235 (\pm 0.014)$
$K \to E$	$0.167 (\pm 0.002)$	$0.169 (\pm 0.004)$	$0.167 (\pm 0.002)$	$0.169 (\pm 0.004)$	$0.145 (\pm 0.002)$
平均值	$0.266 (\pm 0.005)$	$0.249 (\pm 0.005)$	$0.250 (\pm 0.005)$	$0.223 (\pm 0.008)$	$0.209 (\pm 0.009)$

Task	DANN
Task	SO	IWV	DEV	BPDA	TB
$B \to D$	0.228 (±0.003)	0.220 (±0.011)	$0.509 (\pm 0.001)$	$0.233 (\pm 0.053)$	$0.220 (\pm 0.011)$
$B \to E$	$0.322 (\pm 0.009)$	$0.327 (\pm 0.007)$	$0.498 (\pm 0.000)$	$0.313 (\pm 0.081)$	0.235 (±0.017)
$B \to K$	$0.276 (\pm 0.003)$	$0.296 (\pm 0.010)$	$0.272 (\pm 0.013)$	$0.247 (\pm 0.103)$	$0.219 (\pm 0.023)$
$D \to B$	$0.290 (\pm 0.006)$	$0.290 (\pm 0.009)$	$0.253 (\pm 0.045)$	$0.253 (\pm 0.045)$	$0.245 (\pm 0.008)$
$D \to E$	$0.284 (\pm 0.004)$	$0.274 (\pm 0.003)$	$0.299 (\pm 0.007)$	0.252(±0.084)	$0.221 (\pm 0.013)$
$D \to K$	$0.270 (\pm 0.004)$	$0.300 (\pm 0.006)$	$0.303 (\pm 0.007)$	0.217(±0.008)	$0.217 (\pm 0.008)$
$E \to B$	$0.312 (\pm 0.005)$	$0.310 (\pm 0.005)$	$0.312 (\pm 0.005)$	$0.372 (\pm 0.056)$	$0.310 (\pm 0.005)$
$E \to D$	$0.317 (\pm 0.007)$	$0.313 (\pm 0.006)$	$0.313 (\pm 0.006)$	0.327(±0.075)	$0.277 (\pm 0.031)$
$E \to K$	$0.170 (\pm 0.004)$	0.170 (±0.004	$0.170 (\pm 0.004)$	$0.172 (\pm 0.011)$	$0.170 (\pm 0.004)$
$K \to B$	$0.345 (\pm 0.006)$	$0.337 (\pm 0.006)$	0.338 (±0.017)	$0.314 (\pm 0.053)$	$0.314 (\pm 0.053)$
$K \to D$	$0.313 (\pm 0.003)$	$0.360 (\pm 0.005)$	$0.360 (\pm 0.005)$	$0.298 (\pm 0.053)$	$0.296 (\pm 0.019)$
$K \to E$	$0.174 (\pm 0.002)$	0.183 (±0.003)	$0.221 (\pm 0.004)$	$0.194 (\pm 0.057)$	$0.172 (\pm 0.010)$
Avg.	$0.275 (\pm 0.005)$	$0.282 (\pm 0.006)$	$0.321 (\pm 0.010)$	$0.266 (\pm 0.057)$	$0.241 (\pm 0.017)$

任务	DANN
任务	SO	IWV	DEV	BPDA	TB
$B \to D$	0.228 (±0.003)	0.220 (±0.011)	$0.509 (\pm 0.001)$	$0.233 (\pm 0.053)$	$0.220 (\pm 0.011)$
$B \to E$	$0.322 (\pm 0.009)$	$0.327 (\pm 0.007)$	$0.498 (\pm 0.000)$	$0.313 (\pm 0.081)$	0.235 (±0.017)
$B \to K$	$0.276 (\pm 0.003)$	$0.296 (\pm 0.010)$	$0.272 (\pm 0.013)$	$0.247 (\pm 0.103)$	$0.219 (\pm 0.023)$
$D \to B$	$0.290 (\pm 0.006)$	$0.290 (\pm 0.009)$	$0.253 (\pm 0.045)$	$0.253 (\pm 0.045)$	$0.245 (\pm 0.008)$
$D \to E$	$0.284 (\pm 0.004)$	$0.274 (\pm 0.003)$	$0.299 (\pm 0.007)$	0.252(±0.084)	$0.221 (\pm 0.013)$
$D \to K$	$0.270 (\pm 0.004)$	$0.300 (\pm 0.006)$	$0.303 (\pm 0.007)$	0.217(±0.008)	$0.217 (\pm 0.008)$
$E \to B$	$0.312 (\pm 0.005)$	$0.310 (\pm 0.005)$	$0.312 (\pm 0.005)$	$0.372 (\pm 0.056)$	$0.310 (\pm 0.005)$
$E \to D$	$0.317 (\pm 0.007)$	$0.313 (\pm 0.006)$	$0.313 (\pm 0.006)$	0.327(±0.075)	$0.277 (\pm 0.031)$
$E \to K$	$0.170 (\pm 0.004)$	0.170 (±0.004)	$0.170 (\pm 0.004)$	$0.172 (\pm 0.011)$	$0.170 (\pm 0.004)$
$K \to B$	$0.345 (\pm 0.006)$	$0.337 (\pm 0.006)$	0.338 (±0.017)	$0.314 (\pm 0.053)$	$0.314 (\pm 0.053)$
$K \to D$	$0.313 (\pm 0.003)$	$0.360 (\pm 0.005)$	$0.360 (\pm 0.005)$	$0.298 (\pm 0.053)$	$0.296 (\pm 0.019)$
$K \to E$	$0.174 (\pm 0.002)$	0.183 (±0.003)	$0.221 (\pm 0.004)$	$0.194 (\pm 0.057)$	$0.172 (\pm 0.010)$
平均值	$0.275 (\pm 0.005)$	$0.282 (\pm 0.006)$	$0.321 (\pm 0.010)$	$0.266 (\pm 0.057)$	$0.241 (\pm 0.017)$

Task	CMD
Task	SO	IWV	DEV	BPDA	TB
$B \to D$	$0.230 (\pm 0.011)$	0.193 (±0.006)	$0.231 (\pm 0.007)$	0.193 (±0.006)	$0.193 (\pm 0.006)$
$B \to E$	$0.319 (\pm 0.013)$	$0.309 (\pm 0.010)$	$0.308 (\pm 0.009)$	0.218 (±0.011)	$0.218 (\pm 0.011)$
$B \to K$	$0.269 (\pm 0.005)$	$0.230 (\pm 0.007)$	$0.269 (\pm 0.006)$	0.186 (±0.010)	$0.186 (\pm 0.010)$
$D \to B$	$0.290 (\pm 0.015)$	$0.258 (\pm 0.009)$	$0.245 (\pm 0.008)$	$0.228 (\pm 0.006)$	$0.228 (\pm 0.006)$
$D \to E$	$0.280 (\pm 0.009)$	$0.267 (\pm 0.007)$	$0.280 (\pm 0.006)$	$0.203 (\pm 0.007)$	0.203 (±0.007)
$D \to K$	$0.264 (\pm 0.004)$	0.194 (±0.006	0.194 (±0.006)	$0.194 (\pm 0.006)$	0.194 (±0.006)
$E \to B$	$0.314 (\pm 0.009)$	$0.307 (\pm 0.006)$	$0.302 (\pm 0.005)$	$0.279 (\pm 0.010)$	$0.279 (\pm 0.010)$
$E \to D$	$0.320 (\pm 0.020)$	$0.287 (\pm 0.006)$	$0.287 (\pm 0.006)$	$0.258 (\pm 0.014)$	$0.258 (\pm 0.014)$
$E \to K$	$0.174 (\pm 0.013)$	$0.152 (\pm 0.005)$	$0.169 (\pm 0.006)$	$0.139 (\pm 0.005)$	$0.139 (\pm 0.005)$
$K \to B$	$0.346 (\pm 0.022)$	$0.264 (\pm 0.007)$	$0.331 (\pm 0.006)$	$0.264 (\pm 0.007)$	$0.264 (\pm 0.007)$
$K \to D$	$0.314 (\pm 0.013)$	$0.248 (\pm 0.006)$	$0.248 (\pm 0.006)$	$0.248 (\pm 0.006)$	$0.248 (\pm 0.006)$
$K \to E$	$0.178 (\pm 0.007)$	0.147 (±0.004)	$0.178 (\pm 0.007)$	0.147 (±0.004)	$0.147 (\pm 0.004)$
Avg.	$0.275 (\pm 0.012)$	$0.238 (\pm 0.007)$	$0.254 (\pm 0.007)$	$0.213 (\pm 0.008)$	$0.213 (\pm 0.008)$

任务	命令
任务	系统对象	内部权重值	开发	BPDA	TB
$B \to D$	$0.230 (\pm 0.011)$	0.193 (±0.006)	$0.231 (\pm 0.007)$	0.193 (±0.006)	$0.193 (\pm 0.006)$
$B \to E$	$0.319 (\pm 0.013)$	$0.309 (\pm 0.010)$	$0.308 (\pm 0.009)$	0.218 (±0.011)	$0.218 (\pm 0.011)$
$B \to K$	$0.269 (\pm 0.005)$	$0.230 (\pm 0.007)$	$0.269 (\pm 0.006)$	0.186 (±0.010)	$0.186 (\pm 0.010)$
$D \to B$	$0.290 (\pm 0.015)$	$0.258 (\pm 0.009)$	$0.245 (\pm 0.008)$	$0.228 (\pm 0.006)$	$0.228 (\pm 0.006)$
$D \to E$	$0.280 (\pm 0.009)$	$0.267 (\pm 0.007)$	$0.280 (\pm 0.006)$	$0.203 (\pm 0.007)$	0.203 (±0.007)
$D \to K$	$0.264 (\pm 0.004)$	0.194 (±0.006)	0.194 (±0.006)	$0.194 (\pm 0.006)$	0.194 (±0.006)
$E \to B$	$0.314 (\pm 0.009)$	$0.307 (\pm 0.006)$	$0.302 (\pm 0.005)$	$0.279 (\pm 0.010)$	$0.279 (\pm 0.010)$
$E \to D$	$0.320 (\pm 0.020)$	$0.287 (\pm 0.006)$	$0.287 (\pm 0.006)$	$0.258 (\pm 0.014)$	$0.258 (\pm 0.014)$
$E \to K$	$0.174 (\pm 0.013)$	$0.152 (\pm 0.005)$	$0.169 (\pm 0.006)$	$0.139 (\pm 0.005)$	$0.139 (\pm 0.005)$
$K \to B$	$0.346 (\pm 0.022)$	$0.264 (\pm 0.007)$	$0.331 (\pm 0.006)$	$0.264 (\pm 0.007)$	$0.264 (\pm 0.007)$
$K \to D$	$0.314 (\pm 0.013)$	$0.248 (\pm 0.006)$	$0.248 (\pm 0.006)$	$0.248 (\pm 0.006)$	$0.248 (\pm 0.006)$
$K \to E$	$0.178 (\pm 0.007)$	0.147 (±0.004)	$0.178 (\pm 0.007)$	0.147 (±0.004)	$0.147 (\pm 0.004)$
平均值	$0.275 (\pm 0.012)$	$0.238 (\pm 0.007)$	$0.254 (\pm 0.007)$	$0.213 (\pm 0.008)$	$0.213 (\pm 0.008)$

[12] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.

[12] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, 等人. Pytorch: 一种命令式风格的高性能深度学习库. arXiv 预印本 arXiv:1912.01703, 2019.

[13] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830,

[13] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, 等人. Scikit-learn: Python中的机器学习. 机器学习研究杂志, 12:2825-2830,

[15] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.

[15] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, 和 Xiaoqiang Zheng. TensorFlow: 大规模机器学习在异构系统上的应用, 2015. 软件可从 tensorflow.org 获取。

[16] Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul V Buenau, and Motoaki Kawan-abe. Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in Neural Information Processing Systems, pages 1433-1440, 2008.

[16] Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul V Buenau, 和 Motoaki Kawan-abe. 直接重要性估计与模型选择及其在协变量转移适应中的应用. 在《神经信息处理系统进展》中，第 1433-1440 页, 2008。

We study the problem of choosing algorithm hyper-parameters in unsupervised a target domain, drawn from a different input distribution. We follow the strategy to compute several models using different hyper-parameters, and, to subsequently compute a linear aggregation of the models. While several heuristics exist that follow this strategy, methods are still missing that rely on thorough theories for bounding the target error. In this turn, we propose a method that extends weighted least squares to vector-valued functions, e.g., deep neural networks. We show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. We also perform a large electroencephalogram, body sensor signals and signals from mobile phones. Our method

^{1}

outperforms deep embedded validation (DEV) and importance weighted validation (IWV) on all datasets, setting a new state-of-the-art performance for solving parameter choice issues in unsupervised domain adaptation with theoretical error guarantees. We further study several competitive heuristics, all outperforming IWV and DEV on at least five datasets. However, our method outperforms each heuristic on at least five of seven datasets.

我们研究在无监督目标领域中选择算法超参数的问题，该领域来自不同的输入分布。我们遵循的策略是使用不同的超参数计算多个模型，并随后计算模型的线性聚合。虽然存在几种遵循此策略的启发式方法，但仍缺乏依赖于严格理论来界定目标误差的方法。在此基础上，我们提出了一种将加权最小二乘法扩展到向量值函数（例如深度神经网络）的方法。我们表明，所提算法的目标误差渐近上不超过未知最优聚合误差的两倍。我们还进行了大量的脑电图、身体传感器信号和手机信号的实验。我们的方法

^{1}

在所有数据集上均优于深度嵌入验证（DEV）和重要性加权验证（IWV），为无监督领域适应中的参数选择问题设定了新的最先进性能，并提供了理论误差保证。我们进一步研究了几种竞争性的启发式方法，所有这些方法在至少五个数据集上均优于 IWV 和 DEV。然而，我们的方法在至少七个数据集中的五个上优于每种启发式方法。

The goal of unsupervised domain adaptation is to learn a model on unlabeled data from a target input distribution using labeled data from a different source distribution (Pan & Yang, 2010; Ben-David et al., 2010). If this goal is achieved, medical diagnostic systems can successfully be trained on unlabeled images using labeled images with a different modality (Varsavsky et al., 2020; Zou et al., 2020); segmentation models for natural images can be learned using only labeled data from computer simulations Peng et al. (2018); natural language models can be learned from unlabeled biomedical abstracts by means of labeled data from financial journals (Blitzer et al., 2006); industrial quality inspection systems can be learned on unlabeled data from new products using data from related products (Jiao et al., 2019; Zellinger et al., 2020).

无监督领域适应的目标是利用来自不同源分布的标记数据，在目标输入分布的未标记数据上学习模型（Pan & Yang, 2010; Ben-David et al., 2010）。如果这一目标得以实现，医学诊断系统可以成功地在未标记图像上进行训练，使用不同模态的标记图像（Varsavsky et al., 2020; Zou et al., 2020）；自然图像的分割模型可以仅使用来自计算机模拟的标记数据进行学习（Peng et al., 2018）；自然语言模型可以通过来自金融期刊的标记数据，从未标记的生物医学摘要中学习（Blitzer et al., 2006）；工业质量检测系统可以在来自新产品的未标记数据上进行学习，使用来自相关产品的数据（Jiao et al., 2019; Zellinger et al., 2020）。

However, missing target labels combined with distribution shift makes parameter choice a hard problem (Sugiyama et al., 2007; You et al., 2019; Saito et al., 2021; Zellinger et al., 2021; Musgrave et al., 2021). Often, one ends up with a sequence of models, e.g., originating from different hyper-parameter configurations (Ben-David et al., 2007; Saenko et al., 2010; Ganin et al., 2016; Long et al.,

然而，缺失的目标标签与分布转移相结合，使得参数选择成为一个困难的问题（Sugiyama et al., 2007; You et al., 2019; Saito et al., 2021; Zellinger et al., 2021; Musgrave et al., 2021）。通常，最终得到的是一系列模型，例如，源自不同的超参数配置（Ben-David et al., 2007; Saenko et al., 2010; Ganin et al., 2016; Long et al.,

Figure 1: Unsupervised domain adaptation problem (Shimodaira, 2000; Sugiyama et al., 2007; You et al., 2019). Left: Source distribution (solid) and target distribution (dashed). Right: A sequence of different linear models (dashed) is used to find the optimal linear aggregation of the models (solid). Model selection methods (Sugiyama et al., 2007; Kouw et al., 2019; You et al., 2019; Zellinger et al., 2021) cannot outperform the best single model in the sequence, confidence values as used in Zou et al. (2018) are not available, and, approaches based on averages or tendencies of majorities of models (Saito et al., 2017) suffer from a high fraction of large-error-models in the sequence. In contrast, our approach (dotted-dashed) is nearly optimal. In addition, the model computed by our method provably approaches the optimal linear aggregation for increasing sample size. For further

图 1：无监督领域适应问题（Shimodaira, 2000; Sugiyama et al., 2007; You et al., 2019）。左侧：源分布（实线）和目标分布（虚线）。右侧：一系列不同的线性模型（虚线）用于寻找模型的最佳线性聚合（实线）。模型选择方法（Sugiyama et al., 2007; Kouw et al., 2019; You et al., 2019; Zellinger et al., 2021）无法超越序列中最佳单一模型，Zou et al.（2018）中使用的置信值不可用，并且基于模型多数的平均或趋势的方法（Saito et al., 2017）在序列中存在大量高误差模型的比例较高。相比之下，我们的方法（点划线）几乎是最优的。此外，我们的方法计算出的模型在样本量增加时可证明接近最佳线性聚合。有关进一步

Although methods with mathematical error guarantees have been proposed to select the best model in the sequence (Sugiyama et al., 2007; Kouw et al., 2019; You et al., 2019; Zellinger et al., 2021), limited by severe assumptions (cf. Wilson & Cook (2020)). Typical aggregation approaches are (a) to learn an aggregation on source data only (Nozza et al., 2016), (b) to learn an aggregation on a set of (unknown) labeled target examples (Xia et al., 2013; Dai et al., 2007; III & Marcu, 2006; Duan et al., 2012), (c) to learn an aggregation on target examples (pseudo-)labeled based on confidence measures of the given models (Zhou et al., 2021; Ahmed et al., 2022; Sun, 2012; Zou et al., 2018; Saito et al., 2017), (d) to aggregate the models based on data-structure specific transformations (Yang et al., 2012; Ha & Youn, 2021), and, (e) to use specific (possibly not available) knowledge about the given models, such as information obtained at different time-steps of its gradient-based optimization process (French et al., 2018; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Athiwaratkun et al., 2019; Al-Stouhi & Reddy, 2011) or the information that the given models are trained on different (source) distributions (Hoffman et al., 2018; Rakshit et al., 2019; Xu et al., 2018; Kang et al., 2020; Zhang et al., 2015). One problem shared among all methods mentioned above is that they cannot guarantee a small error, even if the sample size grows to infinity. See Figure 1 for a simple illustrative example.

尽管已经提出了具有数学误差保证的方法来选择序列中的最佳模型（Sugiyama et al., 2007; Kouw et al., 2019; You et al., 2019; Zellinger et al., 2021），但受到严苛假设的限制（参见 Wilson & Cook (2020)）。典型的聚合方法包括 (a) 仅在源数据上学习聚合（Nozza et al., 2016），(b) 在一组（未知）标记的目标示例上学习聚合（Xia et al., 2013; Dai et al., 2007; III & Marcu, 2006; Duan et al., 2012），(c) 在基于给定模型的置信度度量对目标示例进行（伪）标记的基础上学习聚合（Zhou et al., 2021; Ahmed et al., 2022; Sun, 2012; Zou et al., 2018; Saito et al., 2017），(d) 基于数据结构特定变换聚合模型（Yang et al., 2012; Ha & Youn, 2021），以及 (e) 使用关于给定模型的特定（可能不可用）知识，例如在其基于梯度的优化过程的不同时间步获得的信息（French et al., 2018; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Athiwaratkun et al., 2019; Al-Stouhi & Reddy, 2011）或关于给定模型在不同（源）分布上训练的信息（Hoffman et al., 2018; Rakshit et al., 2019; Xu et al., 2018; Kang et al., 2020; Zhang et al., 2015）。上述所有方法共同面临的一个问题是，即使样本大小增长到无穷大，也无法保证小误差。有关简单示例，请参见图1。

In this work, we propose (to the best of our knowledge) the first algorithm for computing aggregations of vector-valued models for unsupervised domain adaptation with target error guarantees. We extend the importance weighted least squares algorithm (Shimodaira, 2000) and corresponding recently proposed error bounds (Gizewski et al., 2022) to linear aggregations of vector-valued models. The importance weights are the values of an estimated ratio between target and source density evaluated at the examples. Every method for density-ratio estimation can be used as a basis for our approach, e.g. Sugiyama et al. (2012); Kanamori et al. (2012) and references therein. Our error bound proves that the target error of the computed aggregation is asymptotically at most twice the target error of the optimal aggregation.

在本研究中，我们提出了（据我们所知）第一个用于计算具有目标误差保证的无监督领域适应的向量值模型聚合的算法。我们扩展了重要性加权最小二乘算法（Shimodaira, 2000）及其最近提出的误差界限（Gizewski et al., 2022），以适用于向量值模型的线性聚合。重要性权重是评估在样本处计算的目标和源密度之间的估计比率的值。任何密度比估计方法都可以作为我们方法的基础，例如 Sugiyama et al. (2012); Kanamori et al. (2012) 及其参考文献。我们的误差界限证明了计算的聚合的目标误差在渐近意义上至多是最优聚合目标误差的两倍。

In addition, we perform extensive empirical evaluations on several datasets with academic data (Transformed Moons), text data (Amazon Reviews (Blitzer et al., 2006)), images (MiniDomainNet (Peng et al., 2019; Zellinger et al., 2021)), electroencephalography signals (Sleep-EDF (Eldele et al., 2021; Goldberger et al., 2000)), body sensor signals (UCI-HAR (Anguita et al., 2013), WISDM (Kwapisz et al., 2011)), and, sensor signals from mobile phones and smart watches (HHAR (Stisen et al.,

此外，我们在多个数据集上进行了广泛的实证评估，这些数据集包括学术数据（变换月亮数据集）、文本数据（亚马逊评论（Blitzer et al., 2006））、图像（MiniDomainNet（Peng et al., 2019; Zellinger et al., 2021））、脑电图信号（Sleep-EDF（Eldele et al., 2021; Goldberger et al., 2000））、身体传感器信号（UCI-HAR（Anguita et al., 2013）、WISDM（Kwapisz et al., 2011）），以及来自手机和智能手表的传感器信号（HHAR（Stisen et al.,

We compute aggregations of models obtained from different hyper-parameter settings of 11 domain adaptation methods (e.g., DANN (Ganin et al., 2016) and Deep-Coral Sun & Saenko (2016)). Our method sets a new state of the art for methods with theoretical error guarantees, namely importance weighted validation (IWV) (Sugiyama et al., 2007) and deep embedded validation (DEV) (Kouw et al., 2019), on all datasets. We also study (1) classical least squares aggregation on source data only, (2) majority voting on target predictions, (3) averaging over model confidences, and (4) learning based on pseudo-labels. All of these heuristics outperform IWV and DEV on at least five of seven datasets, which is a result of independent interest. In contrast, our method outperforms each heuristic on at least five of seven datasets.

我们计算了来自11种领域适应方法（例如，DANN (Ganin et al., 2016) 和 Deep-Coral Sun & Saenko (2016)）不同超参数设置获得的模型的聚合。我们的方法在所有数据集上为具有理论误差保证的方法设定了新的最先进水平，即重要性加权验证（IWV）(Sugiyama et al., 2007) 和深度嵌入验证（DEV）(Kouw et al., 2019)。我们还研究了（1）仅在源数据上进行的经典最小二乘聚合，（2）在目标预测上的多数投票，（3）基于模型置信度的平均，以及（4）基于伪标签的学习。所有这些启发式方法在至少七个数据集中的五个上都优于IWV和DEV，这本身就是一个独立的研究兴趣。相比之下，我们的方法在至少七个数据集中的五个上优于每个启发式方法。

It is well known that aggregations of models in an ensemble often outperform individual models (Dong of aggregation are Boosting (Schapire, 1990; Breiman, 1998), Bootstrap Aggregating (bagging) (Breiman, 1994; 1996a) and Stacking (Wolpert, 1992; Breiman, 1996b). For example, averages of multiple models pre-trained on data from a distribution different from the target one have recently been shown to achieve state-of-the-art performance on ImageNet (Wortsman et al., 2022) and their good generalization properties can be related to flat minima (Hochreiter & Schmidhuber, 1994; 1997). However, most such methods don't take into account a present distribution shift.

众所周知，集成中的模型聚合通常优于单个模型（聚合的例子有提升（Boosting）（Schapire, 1990; Breiman, 1998）、自助聚合（Bootstrap Aggregating，bagging）（Breiman, 1994; 1996a）和堆叠（Stacking）（Wolpert, 1992; Breiman, 1996b）。例如，最近的研究表明，在与目标分布不同的数据上预训练的多个模型的平均值在 ImageNet 上达到了最先进的性能（Wortsman et al., 2022），而它们良好的泛化特性可以与平坦极小值相关联（Hochreiter & Schmidhuber, 1994; 1997）。然而，大多数此类方法并未考虑当前的分布变化。

Although some ensemble learning methods exist, which take into account a present distribution shift, in contrast to our work, they are either relying on labeled target data (Nozza et al., 2016; Xia et al., 2013; III & Marcu, 2006; Dai et al., 2007; Mayr et al., 2016), are restricted by fixing the aggregation weights to be the same (Razar & Samothrakis, 2019), make assumptions on the models in the sequence or the corresponding process for learning the models (Yang et al., 2012; Ha & Youn, 2021; French et al., 2018; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Athiwaratkun et al., 2019; Al-Stouhi & Reddy, 2011; Hoffman et al., 2018; Rakshit et al., 2019; Xu et al., 2018; Kang et al., 2020; Zhang et al., 2015), or, learn an aggregation based on the heuristic approach of (pseudo-)labeling some target data based on confidence measures of models in the sequence (Zhou et al., 2021; Ahmed et al., 2022; Sun, 2012; Zou et al., 2018; Saito et al., 2017). Another crucial difference of all methods above is that none of these methods can guarantee a small target error in the general setting (distribution shift, vector valued models, different classes, single source domain) described above, even if the sample size grows to infinity.

尽管存在一些考虑当前分布变化的集成学习方法，但与我们的工作相比，它们要么依赖于标记的目标数据（Nozza et al., 2016; Xia et al., 2013; III & Marcu, 2006; Dai et al., 2007; Mayr et al., 2016），要么限制聚合权重相同（Razar & Samothrakis, 2019），对序列中的模型或学习模型的相应过程做出假设（Yang et al., 2012; Ha & Youn, 2021; French et al., 2018; Laine & Aila, 2017; Tarvainen & Valpola, 2017; Athiwaratkun et al., 2019; Al-Stouhi & Reddy, 2011; Hoffman et al., 2018; Rakshit et al., 2019; Xu et al., 2018; Kang et al., 2020; Zhang et al., 2015），或者基于序列中模型的置信度度量，学习基于（伪）标记某些目标数据的启发式方法的聚合（Zhou et al., 2021; Ahmed et al., 2022; Sun, 2012; Zou et al., 2018; Saito et al., 2017）。上述所有方法的另一个关键区别在于，即使样本量增长到无穷大，这些方法也无法保证在上述一般设置（分布变化、向量值模型、不同类别、单一源领域）下的小目标误差。

Another branch of research are methods which aim at selecting the best model in the sequence. Although, such methods with error bounds have been proposed for the general setting above (Sugiyama et al., 2007; You et al., 2019; Zellinger et al., 2021), they cannot overcome a limited performance of the best model in the given sequence (cf. Figure 1 and Section 6 in the Supplementary Material of Zellinger et al. (2021)). In contrast, our method can outperform the best model in the sequence, and our empirical evaluations show that this is indeed the case in practical examples. A recent kernel-based algorithm for univariate regression, that is similar to ours, can be found in Gizewski et al. (2022). However, in contrast to Gizewski et al. (2022), our method allows a much more general form of vector-valued models which are not necessarily obtained from regularized kernel least squares, and, can therefore be applied to practical deep learning tasks.

另一研究分支是旨在选择序列中最佳模型的方法。尽管已经为上述一般设置提出了具有误差界限的方法（Sugiyama et al., 2007; You et al., 2019; Zellinger et al., 2021），但它们无法克服给定序列中最佳模型的有限性能（参见 Zellinger et al. (2021) 的图 1 和补充材料第 6 节）。相反，我们的方法能够超越序列中的最佳模型，我们的实证评估表明，在实际例子中确实如此。最近的一种基于核的单变量回归算法，类似于我们的算法，可以在 Gizewski et al. (2022) 中找到。然而，与 Gizewski et al. (2022) 相比，我们的方法允许一种更一般形式的向量值模型，这些模型不一定来自正则化的核最小二乘，因此可以应用于实际的深度学习任务。

Our work employs technical tools developed in Caponnetto & De Vito (2007; 2005). In fact, we extend Caponnetto & De Vito (2007; 2005) to deal with importance weighted least squares. Finally, it is important to note Huang et al. (2006), where a core Lemma of our proofs is proposed.

我们的工作采用了 Caponnetto & De Vito (2007; 2005) 中开发的技术工具。事实上，我们扩展了 Caponnetto & De Vito (2007; 2005) 以处理重要性加权最小二乘。最后，值得注意的是 Huang et al. (2006)，其中提出了我们证明的核心引理。

Notation and Setup Let

X \subset R^{d_{1}}

be a compact input space and

Y \subset R^{d_{2}}

be a compact label space with inner product

⟨ ., . ⟩_{V}

,such that for the associated norm

∥ y ∥_{V} \leq y_{0}

holds for all

y \in Y

and some

y_{0} > 0

. Following Ben-David et al. (2010),we consider two datasets: A source dataset

(x, y) = ((x_{1}, y_{1}), \dots, (x_{n}, y_{n})) \in {(X \times Y)}^{n}

independently drawn according to some source distribution (probability measure)

p

X \times Y

and an unlabeled target dataset

x^{'} = (x_{1}^{'}, \dots, x_{m}^{'}) \in

X^{m}

with elements independently drawn according to the marginal distribution

^{2} q_{X}

of some target further denote by

R_{q} (f) = \int_{X \times V} ∥ f (x) - y ∥_{Y}^{2} d q (x, y)

the expected target risk of a vector valued function

f : X \to Y

w.r.t. the least squares loss.

记号和设置设

X \subset R^{d_{1}}

为一个紧凑的输入空间，

Y \subset R^{d_{2}}

为一个紧凑的标签空间，具有内积

⟨ ., . ⟩_{V}

，使得对于关联的范数

∥ y ∥_{V} \leq y_{0}

，对所有

y \in Y

和某些

y_{0} > 0

都成立。根据 Ben-David 等人（2010）的研究，我们考虑两个数据集：一个源数据集

(x, y) = ((x_{1}, y_{1}), \dots, (x_{n}, y_{n})) \in {(X \times Y)}^{n}

，根据某个源分布（概率测度）

p

在

X \times Y

上独立抽取，和一个未标记的目标数据集

x^{'} = (x_{1}^{'}, \dots, x_{m}^{'}) \in

X^{m}

，其元素根据某个目标的边际分布

^{2} q_{X}

独立抽取，进一步用

R_{q} (f) = \int_{X \times V} ∥ f (x) - y ∥_{Y}^{2} d q (x, y)

表示向量值函数

f : X \to Y

关于最小二乘损失的期望目标风险。

Main Assumptions We rely (a) on the covariate shift assumption that the source conditional distribution

p (y ∣ x)

equals the target conditional distribution

q (y ∣ x)

,and,(b) on the bounded density ratio assumption that there is a function

β : X \to [0, B]

with

B > 0

such that

d q_{X} (x) = β (x) d p_{X} (x)

主要假设我们依赖于 (a) 协变量转移假设，即源条件分布

p (y ∣ x)

等于目标条件分布

q (y ∣ x)

，以及 (b) 有界密度比假设，即存在一个函数

β : X \to [0, B]

，使得

B > 0

，并且

d q_{X} (x) = β (x) d p_{X} (x)

。

for the regression functions given by

f_{q} (x) = \int_{V} y d q {(y ∣ x)}^{3}

,see e.g. Cucker &Smale (2002, Proposition 1). Unfortunately,the right hand side of Eq. (1) contains information about labels

f_{q} (x)

which are not given in our setting of unsupervised domain adaptation. However, borrowing an idea from importance sampling, it is possible to estimate Eq. (1). More precisely, from the covariate shift assumption we get

f_{p} (x) = \int_{Y} y d p (y ∣ x) = f_{q} (x)

and we can use the bounded density ratio

β

to obtain

对于由

f_{q} (x) = \int_{V} y d q {(y ∣ x)}^{3}

给出的回归函数，参见 Cucker & Smale (2002, 命题 1)。不幸的是，方程 (1) 的右侧包含关于标签

f_{q} (x)

的信息，而这些信息在我们的无监督领域适应设置中并未给出。然而，借用重要性抽样的一个想法，可以估计方程 (1)。更准确地说，从协变量转移假设中我们得到

f_{p} (x) = \int_{Y} y d p (y ∣ x) = f_{q} (x)

，并且我们可以使用有界密度比

β

来获得

which extends importance weighted least squares (Shimodaira, 2000; Kanamori et al., 2009) to linear aggregations

\sum_{i = 1}^{l} c_{i} f_{i}

of vector-valued functions

f_{1}, \dots, f_{l}

. The unique minimizer of Eq. (2) can be approximated based on available data analogously to classical least squares estimation as detailed in Algorithm 1. In the following, we call Algorithm 1 Importance Weighted Least Squares Linear Aggregation (IWA).

这扩展了重要性加权最小二乘法 (Shimodaira, 2000; Kanamori et al., 2009) 到向量值函数

f_{1}, \dots, f_{l}

的线性聚合

\sum_{i = 1}^{l} c_{i} f_{i}

。方程 (2) 的唯一最小化器可以基于可用数据类似于经典最小二乘估计进行近似，如算法 1 中详细说明的那样。在下面，我们称算法 1 为重要性加权最小二乘线性聚合 (IWA)。

Let us start by introducing some further notation:

L^{2} (p)

refers to the Lebesgue-Bochner space of functions from

X

Y

,associated to a measure

p

X

with corresponding inner product

⟨ \cdot, \cdot ⟩_{L^{2} (p)}

(this space basically consists of all

Y

-valued functions whose

Y

-norms are square integrable with respect to the given measure

p

). Moreover,let us introduce the (positive semi-definite) Gram matrix

G = {({⟨ f_{i}, f_{j} ⟩}_{L^{2} (q X)})}_{i, j = 1}^{l}

and the vector

\bar{g} = {({⟨ β f_{p}, f_{i} ⟩}_{L^{2} (p X)})}_{i = 1}^{l}

. We can assume that

G

is invertible (and thus positive definite), since otherwise some models are too similar to others and can

c^{*} = (c_{1}^{*}, \dots, c_{l}^{*}) = G^{- 1} \bar{g}

,see Lemma 4 .

让我们开始引入一些进一步的符号：

L^{2} (p)

指的是从

X

到

Y

的 Lebesgue-Bochner 函数空间，关联到

p

上的测度，具有相应的内积

⟨ \cdot, \cdot ⟩_{L^{2} (p)}

（该空间基本上由所有

Y

值的函数组成，这些函数的

Y

范数相对于给定的测度

p

是平方可积的）。此外，让我们引入（正半定）Gram 矩阵

G = {({⟨ f_{i}, f_{j} ⟩}_{L^{2} (q X)})}_{i, j = 1}^{l}

和向量

\bar{g} = {({⟨ β f_{p}, f_{i} ⟩}_{L^{2} (p X)})}_{i = 1}^{l}

。我们可以假设

G

是可逆的（因此是正定的），因为否则某些模型与其他模型过于相似，可以

c^{*} = (c_{1}^{*}, \dots, c_{l}^{*}) = G^{- 1} \bar{g}

，见引理 4。

However,neither

G

nor the vector

\bar{g}

is accessible in practice,because there is no access to the target measure

q_{X}

. Driven by the law of large numbers we try to approximate them by averages over our given data and therefore arrive at the formulas for

\tilde{G}

and

\tilde{g}

given in Algorithm 1. This leads to the approximation

\tilde{f}

. Up to this point,we were only considering an intuitive perspective on the problem setting,therefore,we will now formally discuss statements on the distance between the model

\tilde{f}

and the optimal linear model

f^{*} = \sum_{i = 1}^{l} c_{i}^{*} f_{i}

,measured in terms of target risks,and how this distance behaves with increasing sample sizes. This is what we attempt with our main result:

然而，实际上既没有

G

也没有向量

\bar{g}

是可获取的，因为无法访问目标度量

q_{X}

。在大数法则的驱动下，我们试图通过对给定数据的平均值来近似它们，因此得出了算法 1 中给出的

\tilde{G}

和

\tilde{g}

的公式。这导致了近似

\tilde{f}

。到目前为止，我们只考虑了问题设置的直观视角，因此，我们现在将正式讨论模型

\tilde{f}

与最佳线性模型

f^{*} = \sum_{i = 1}^{l} c_{i}^{*} f_{i}

之间的距离的陈述，该距离是通过目标风险来衡量的，以及随着样本量的增加，这个距离的变化情况。这是我们主要结果所尝试的内容：

Before we give an outline of the proof (see Section A), let us briefly comment on the main message of Algorithm 1. Observe,that (Cucker &Smale,2002,Proposition 1)

R_{q} (f) - R_{q} (f_{a}) =

{‖ \begin{matrix} f - f_{q} \end{matrix} ‖}_{L^{2} (q_{X})}^{2}

can be interpreted as the total target error made by Algorithm 1,sometimes called function and the excess risk equals the target error of Ben-David et al. (2010). Eq. (4) compares this error for the aggregation

\tilde{f}

,computed by Algorithm 1,to the error for the optimal aggregation

f^{*}

. Note that the error of the optimal aggregation

f^{*}

is unavoidable in the sense that it is determined by the decision of searching for linear aggregations of

f_{1}, \dots, f_{l}

only. However,if the models

f_{1}, \dots, f_{l}

are sufficiently different, then this error can be expected to be small. Theorem 1 tells us that the error of

\tilde{f}

approaches the one of

f^{*}

with increasing target and source sample size. The rate of convergence is at least linear. Finally, we emphasize that Theorem 1 does not take into account the error of the density-ratio estimation. We refer to the recent work Gizewski et al. (2022), who, for the first time, included such error in the analysis of importance weighted least squares.

在我们给出证明的概要之前（见 A 节），让我们简要评论一下算法 1 的主要信息。请注意，(Cucker & Smale, 2002, 命题 1)

R_{q} (f) - R_{q} (f_{a}) =

{‖ \begin{matrix} f - f_{q} \end{matrix} ‖}_{L^{2} (q_{X})}^{2}

可以被解释为算法 1 所产生的总目标误差，有时称为函数，而超额风险等于 Ben-David 等人（2010）的目标误差。公式 (4) 比较了算法 1 计算的聚合误差

\tilde{f}

与最优聚合的误差

f^{*}

。请注意，最优聚合的误差

f^{*}

是不可避免的，因为它仅由寻找

f_{1}, \dots, f_{l}

的线性聚合的决策所决定。然而，如果模型

f_{1}, \dots, f_{l}

之间的差异足够大，则可以预期该误差会很小。定理 1 告诉我们，随着目标和源样本大小的增加，误差

\tilde{f}

接近于

f^{*}

的误差。收敛速率至少是线性的。最后，我们强调定理 1 并未考虑密度比估计的误差。我们提到最近的工作 Gizewski 等人（2022），他们首次在重要性加权最小二乘的分析中包含了这样的误差。

Let us now give a brief outline for the proof of Theorem 1. One key part concerns the existence of a Hilbert space

H

with associated inner product

⟨ ., \cdot ⟩_{H}

(a reproducing kernel space of functions from

X \to Y

) which contains all given models

f_{1}, \dots, f_{l}

and the regression function

f_{q} = f_{p}

. The space

H

can be constructed from any given models that are bounded and continuous functions. Furthermore, Algorithm 1 does not need any knowledge of

H

,which is a modeling assumption only needed for the proofs, so that we can apply many arguments developed in Caponnetto & De Vito (2007; 2005).

H

is also not necessarily generated by a prescribed kernel such as Gaussian or linear kernel, and, no further smoothness assumption is required, see Sections A and B in the Supplementary Material.

现在让我们简要概述定理 1 的证明。一个关键部分涉及到存在一个希尔伯特空间

H

，其相关的内积

⟨ ., \cdot ⟩_{H}

（来自

X \to Y

的函数的再生核空间），该空间包含所有给定的模型

f_{1}, \dots, f_{l}

和回归函数

f_{q} = f_{p}

。空间

H

可以从任何给定的有界且连续的函数模型构建。此外，算法 1 不需要对

H

有任何了解，这只是一个仅在证明中需要的建模假设，因此我们可以应用 Caponnetto 和 De Vito（2007; 2005）中发展出的许多论点。

H

也不一定由规定的核生成，例如高斯核或线性核，并且不需要进一步的光滑性假设，详见补充材料中的 A 和 B 节。

Moreover,in this setting one can express the excess risk as follows:

R_{q} (f) - R_{q} (f_{q}) =

{‖ \begin{matrix} A (f - f_{q}) \end{matrix} ‖}_{H}^{2}

for some bounded linear operator

A : H \to H

. This also allows us to formulate the entries of

G

and

\bar{g}

in terms of the inner product

⟨ \cdot, \cdot ⟩_{H}

instead. Using properties related to the operators that appear in the construction of

H

,in combination with Hoeffding-like concentration bounds in Hilbert spaces and bounds that measure, e.g., the deviation between empirical averages in source and target domain (as done in Gretton et al. (2006, Lemma 4)), we can quantify differences between the entries of

G

and

\tilde{G}

(and

\bar{g}

and

\tilde{g}

respectively) in terms of

n, m

and

δ

. This leads to Eq. (4).

此外，在这种情况下，可以将超额风险表示为：

R_{q} (f) - R_{q} (f_{q}) =

{‖ \begin{matrix} A (f - f_{q}) \end{matrix} ‖}_{H}^{2}

对于某些有界线性算子

A : H \to H

。这也使我们能够将

G

和

\bar{g}

的条目以内积

⟨ \cdot, \cdot ⟩_{H}

的形式进行表述。利用与

H

的构造中出现的算子相关的性质，结合希尔伯特空间中的 Hoeffding 类浓缩界限以及测量源域和目标域中经验平均值之间偏差的界限（如 Gretton 等人（2006，引理 4）所做），我们可以量化

G

和

\tilde{G}

（以及

\bar{g}

和

\tilde{g}

）的条目之间的差异，以

n, m

和

δ

为依据。这导致了方程 (4)。

We now empirically evaluate the performance of our approach compared to classical ensemble learning baselines and state-of-the-art model selection methods. Therefore, we structure our empirical evaluation as follows. First, we outline our experimental setup for unsupervised domain adaptation and introduce all domain adaptation methods for our analysis. Second, we describe the ensemble learning and model selection baselines, and third, we present the datasets used for our experiments. We then conclude with our results and a detailed discussion thereof.

我们现在对我们的方法的性能进行实证评估，与经典的集成学习基线和最先进的模型选择方法进行比较。因此，我们将我们的实证评估结构化如下。首先，我们概述无监督领域适应的实验设置，并介绍我们分析的所有领域适应方法。其次，我们描述集成学习和模型选择基线，第三，我们展示用于实验的数据集。最后，我们总结我们的结果并进行详细讨论。

Figure 2: Top: Mean classification accuracy (y-axis) of our method (IWA), source-only regression (SOR), deep embedded validation (DEV) and individual models (green: source accuracy, orange: target accuracy) used in the aggregation for the HHAR dataset (Stisen et al., 2015) over 3 seeds. The individual models (x-axis) are trained with DIRT (Shu et al., 2018) for different hyper-parameter choices. Bottom: Scaled Aggregation weights (y-axis) for individual models (x-axis) computed by IWA, SOR and DEV (average over 3 seeds). Instead of searching for the best model in the sequence, IWA effectively uses all models in the sequence and obtains a performance not reachable by any procedure selecting only one model.

图2：顶部：我们的方法（IWA）、仅源回归（SOR）、深度嵌入验证（DEV）和用于HHAR数据集（Stisen等，2015）的聚合中的单个模型（绿色：源准确率，橙色：目标准确率）的平均分类准确率（y轴），基于3个种子的结果。单个模型（x轴）使用不同超参数选择的DIRT（Shu等，2018）进行训练。底部：由IWA、SOR和DEV计算的单个模型（x轴）的缩放聚合权重（y轴）（3个种子的平均值）。IWA有效地利用序列中的所有模型，而不是仅仅搜索序列中的最佳模型，从而获得任何仅选择一个模型的程序无法达到的性能。

To assess the performance of our ensemble learning Algorithm 1 IWA, we perform numerous experiments with different domain adaptation algorithms on different datasets. By changing the hyper-parameters of each algorithm, we obtain, as results of applying these algorithms, sequences of models. The goal of our method is to find optimal models based on combinations of candidates from each sequence. As domain adaptation algorithms, we consider the AdaTime benchmark suite, and run our experiments on language, image, text and time-series data. This suite comprises a collection of 11 domain adaptation algorithms. We follow their evaluation setup and apply the following algorithms: Adversarial Spectral Kernel Matching (AdvSKM) (Liu & Xue, 2021), Deep Domain Confusion (DDC) (Tzeng et al., 2014), Correlation Alignment via Deep Neural Networks (Deep-Coral) (Sun et al., 2017), Central Moment Discrepancy (CMD) (Zellinger et al., 2017), Higher-order Moment Matching (HoMM) (Chen et al., 2020), Minimum Discrepancy Estimation for Deep Domain Adaptation (MMDA) (Rahman et al., 2020), Deep Subdomain Adaptation (DSAN) (Zhu et al., 2021), Domain-Adversarial Neural Networks (DANN) (Ganin et al., 2016), Conditional Adversarial Domain Adaptation (CDAN) (Long et al., 2018), A DIRT-T Approach to Unsupervised Domain Adaptation (DIRT) (Shu et al., 2018) and Convolutional deep Domain Adaptation model for Time-Series data (CoDATS) (Wilson et al., 2020). In addition to the sequence of models, IWA requires an estimate of the density ratio between source and target domain. To compute this quantity we follow (Bickel et al., 2007) and (You et al., 2019, Section 4.3), and, train a classifier discriminating between source and target data. The output of this classifier is then used to approximate the density ratio denoted as

β

in Algorithm 1. Overall, to compute the results in our tables we trained 16680 models over approximately a timeframe of 1500 GPU/hours using computation resources of NVIDIA P100 16GB GPUs.

为了评估我们的集成学习算法 IWA 的性能，我们在不同数据集上使用不同的领域适应算法进行了大量实验。通过改变每个算法的超参数，我们获得了应用这些算法的模型序列。我们方法的目标是基于每个序列中候选模型的组合找到最佳模型。作为领域适应算法，我们考虑了 AdaTime 基准套件，并在语言、图像、文本和时间序列数据上运行我们的实验。该套件包含 11 种领域适应算法的集合。我们遵循他们的评估设置，并应用以下算法：对抗谱核匹配 (AdvSKM) (Liu & Xue, 2021)、深度领域混淆 (DDC) (Tzeng et al., 2014)、通过深度神经网络的相关性对齐 (Deep-Coral) (Sun et al., 2017)、中心矩差异 (CMD) (Zellinger et al., 2017)、高阶矩匹配 (HoMM) (Chen et al., 2020)、深度领域适应的最小差异估计 (MMDA) (Rahman et al., 2020)、深度子领域适应 (DSAN) (Zhu et al., 2021)、领域对抗神经网络 (DANN) (Ganin et al., 2016)、条件对抗领域适应 (CDAN) (Long et al., 2018)、无监督领域适应的 DIRT-T 方法 (DIRT) (Shu et al., 2018) 和时间序列数据的卷积深度领域适应模型 (CoDATS) (Wilson et al., 2020)。除了模型序列，IWA 还需要估计源域和目标域之间的密度比。为了计算这个量，我们遵循 (Bickel et al., 2007) 和 (You et al., 2019, 第 4.3 节)，并训练一个分类器来区分源数据和目标数据。该分类器的输出用于近似在算法 1 中表示的密度比

β

。总体而言，为了计算我们表中的结果，我们在大约 1500 GPU/小时的时间框架内训练了 16680 个模型，使用的计算资源为 NVIDIA P100 16GB GPU。

For example, consider the top plot of Figure 2, where we compare the performance of Algorithm 1 to deep embedded validation (DEV) (You et al., 2019), a heuristics baseline source-only regression (SOR, see Section 5.2) and each individual model in the sequence. The bottom plot shows the the scaled aggregation weights, i.e. how much each individual model contributes to the aggregated prediction of IWA, DEV, and SOR. In this example, the given sequence of models is obtained from applying the algorithm proposed in Shu et al. (2018) with different hyper-parameter choices to the Heterogeneity Human Activity Recognition dataset (Stisen et al., 2015). See Section D. 3 in the Supplementary Material for the exact hyper-parameter values.

例如，考虑图2的顶部图表，我们将算法1的性能与深度嵌入验证（DEV）（You et al., 2019）、启发式基线源仅回归（SOR，见第5.2节）以及序列中的每个单独模型进行比较。底部图表显示了缩放的聚合权重，即每个单独模型对IWA、DEV和SOR的聚合预测的贡献。在这个例子中，给定的模型序列是通过对Heterogeneity Human Activity Recognition数据集（Stisen et al., 2015）应用Shu et al.（2018）提出的算法，并选择不同的超参数得到的。有关确切的超参数值，请参见补充材料中的第D.3节。

Heuristic Baselines The first baseline is majority voting on target data (TMV). It aggregates the predictions of all models by counting the overall class predictions and selects the class with the maximum prediction count as ensemble output. In addition, we implement three heuristic baselines which aggregate the vector-valued output, i.e. probabilities, of all classifiers using weights learned via linear regression. The final ensemble prediction is then made by selecting the class with the highest probability. The three heuristic regression baselines differ in the input used for the performed regression. Source-only regression (SOR) trains a regression model on classifier predictions (of the given models) and labels from the source domain only. Target majority voting regression (TMR) uses the same voting procedure as explained above to generate pseudo-labels on the target domain, which are then further used to train a linear regression model. In contrast, target confidence average regression (TCR) selects the highest average class probability over all classifiers to pseudo-label the target samples, which is then used for training the linear regression model.

启发式基线第一个基线是对目标数据的多数投票（TMV）。它通过计算所有模型的整体类别预测来聚合预测，并选择具有最大预测计数的类别作为集成输出。此外，我们实现了三个启发式基线，这些基线通过线性回归学习的权重聚合所有分类器的向量值输出，即概率。最终的集成预测是通过选择具有最高概率的类别来完成的。这三个启发式回归基线在执行回归时使用的输入不同。仅源回归（SOR）仅在源领域的分类器预测（给定模型的预测）和标签上训练回归模型。目标多数投票回归（TMR）使用上述相同的投票程序在目标领域生成伪标签，然后进一步用于训练线性回归模型。相反，目标置信度平均回归（TCR）选择所有分类器中最高的平均类别概率来对目标样本进行伪标签，然后用于训练线性回归模型。

Baselines with Theoretical Error Guarantees We compare IWA to the model selection methods importance weighted validation (IWV) (Sugiyama et al., 2007) and deep embedded validation (DEV) (You et al., 2019), which select models according to their (importance weighted) target risk. Both methods assume the knowledge of an estimated density ratio between target and source domains. In our experiments we follow Bickel et al. (2007); You et al. (2019) and estimate this ratio, by using a classifier that discriminates between source and target domain (see Supplementary Material Section D for more details).

具有理论误差保证的基线我们将IWA与模型选择方法进行比较，包括重要性加权验证（IWV）（Sugiyama等，2007）和深度嵌入验证（DEV）（You等，2019），这两种方法根据其（重要性加权的）目标风险选择模型。这两种方法假设知道目标和源领域之间的估计密度比。在我们的实验中，我们遵循Bickel等（2007）；You等（2019），通过使用一个能够区分源领域和目标领域的分类器来估计这个比率（有关更多细节，请参见补充材料D节）。

We evaluate the previously mentioned methods according to a diverse set of datasets, including language, image and time-series data. All datasets have a train, evaluation and test split, with results only presented on the held-out test sets. For additional details we refer to Appendix C and D.

我们根据多样化的数据集评估之前提到的方法，包括语言、图像和时间序列数据。所有数据集都有训练、评估和测试的划分，结果仅在保留的测试集上呈现。有关更多详细信息，请参阅附录 C 和 D。

AmazonReviews is based on Blitzer et al. (2006) and consists of text reviews from four domains: books, DVDs, electronics, and kitchen appliances. Reviews are encoded in feature vectors of bag-of-words unigrams and bigrams with binary labels indicating the rankings. From the four categories we as target domain.

AmazonReviews 基于 Blitzer 等人（2006）的研究，包含来自四个领域的文本评论：书籍、DVD、电子产品和厨房电器。评论以词袋 unigram 和 bigram 的特征向量进行编码，并带有指示排名的二进制标签。我们将这四个类别作为目标领域。

UCI-HAR The Human Activity Recognition (Anguita et al., 2013) dataset from the UC Irvine Repository contains data from three motion sensors (accelerometer, gyroscope and body-worn categories, namely, walking, walking upstairs, downstairs, standing, sitting, and lying down.

UCI-HAR 人体活动识别（Anguita 等人，2013）数据集来自加州大学欧文分校库，包含来自三个运动传感器（加速度计、陀螺仪和身体佩戴类别）的数据，即步行、上楼、下楼、站立、坐着和躺下。

Sleep-EDF The Sleep Stage Classification time-series setting aims to classify the electroencephalography (EEG) signals into five stages i.e., Wake (W), Non-Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM). Analogous to Ragab et al. (2022); Eldele et al. (2021), we adopt the Sleep-EDF-20 dataset obtained from PhysioBank (Goldberger et al., 2000), which contains EEG readings from 20 healthy subjects.

Sleep-EDF 睡眠阶段分类时间序列设置旨在将脑电图 (EEG) 信号分类为五个阶段，即清醒 (W)、非快速眼动阶段 (N1, N2, N3) 和快速眼动 (REM)。与 Ragab et al. (2022); Eldele et al. (2021) 类似，我们采用从 PhysioBank (Goldberger et al., 2000) 获得的 Sleep-EDF-20 数据集，该数据集包含 20 名健康受试者的 EEG 读数。

We separate the applied methods into two groups, namely heuristic and methods with theoretical error guarantees. All tables show accuracies of source-only (SO) and target-best (TB) models, where source-only denotes training without domain adaptation and target-best the best performing model obtained among all parameter settings. We highlight in bold the performance of the best performing method with theoretical error guarantees, and in italic the best performing heuristic. See Table 1 for results. Please find the full tables in the Supplementary Material Section D.

我们将应用的方法分为两组，即启发式方法和具有理论误差保证的方法。所有表格显示源仅 (SO) 和目标最佳 (TB) 模型的准确性，其中源仅表示不进行领域适应的训练，目标最佳表示在所有参数设置中获得的最佳表现模型。我们用粗体突出显示具有理论误差保证的最佳表现方法的性能，用斜体表示最佳表现的启发式方法。请参见表 1 以获取结果。完整表格请参见补充材料 D 部分。

Outperformance of heuristics: It is interesting to note that each heuristic outperforms IWV and DEV on at least five of seven datasets. Moreover, every heuristic outperforms the (average) target best model (TB) in at least two cases, making it impossible for any model selection method to win in these cases. These facts highlight the quality of the predictions of our chosen heuristics. However, each heuristic is outperformed by our method on at least five of seven datasets.

启发式算法的优越性：有趣的是，每个启发式算法在七个数据集中的至少五个数据集上都优于 IWV 和 DEV。此外，每个启发式算法在至少两个案例中优于（平均）目标最佳模型（TB），这使得任何模型选择方法在这些案例中都无法获胜。这些事实突显了我们选择的启发式算法预测的质量。然而，我们的方法在七个数据集中的至少五个数据集上优于每个启发式算法。

Information in aggregation weights and robustness w.r.t. inaccurate models: It is interesting to observe that, in contrast to the other heuristic aggregation baselines, the aggregation weights

c_{1}, \dots, c_{l}

of our method tend to be larger for accurate models,see Section D.5. Another result is that our method tends to be less sensitive to a high number of inaccurate models than the baselines, see Section D.6. This serves as another reason for its high empirical performance.

聚合权重中的信息及其对不准确模型的鲁棒性：有趣的是，与其他启发式聚合基线相比，我们方法的聚合权重

c_{1}, \dots, c_{l}

对于准确模型往往较大，见第 D.5 节。另一个结果是，我们的方法对大量不准确模型的敏感性通常低于基线，见第 D.6 节。这为其高实证性能提供了另一个理由。

We present a constructive theory-based method for approaching parameter choice issues in the setting of unsupervised domain adaptation. Its theoretical approach relies on the extension of weighted least squares to vector-valued functions. The resulting aggregation method distinguishes itself by a wide trees and neural networks. A broad empirical comparative study on benchmark datasets for language,

我们提出了一种基于构造理论的方法，以解决无监督领域适应中的参数选择问题。其理论方法依赖于加权最小二乘法扩展到向量值函数。由此产生的聚合方法通过广泛的树和神经网络而独树一帜。对语言的基准数据集进行的广泛实证比较研究，

images, body sensor signals and handy signals, underpins the theory-based optimality claim. It is left for future research to further refine the theory and its estimates, e.g., by exploiting concentration bounds from Gretton et al. (2006) or advanced density ratio estimators from Sugiyama et al. (2012).

图像、身体传感器信号和手持信号，支持了基于理论的最优性主张。未来的研究将进一步完善理论及其估计，例如，通过利用 Gretton 等人（2006）的集中界限或 Sugiyama 等人（2012）的高级密度比估计器。

Method	Amazon Reviews
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.769 (\pm 0.009)$	$0.777 (\pm 0.010)$	$0.778 (\pm 0.010)$	0.778(±0.011)	$0.777 (\pm 0.010)$	$0.765 (\pm 0.011)$	$0.766 (\pm 0.011)$	$0.778 (\pm 0.010)$	$0.769 (\pm 0.012)$
AdvSKM	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009$	$0.779 (\pm 0.010)$	$0.779 (\pm 0.008)$	$0.778 (\pm 0.011)$	$0.769 (\pm 0.012)$	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009)$	$0.770 (\pm 0.012)$
DIRT	$0.764 (\pm 0.009)$	$0.786 (\pm 0.008)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.008)$	$0.800 (\pm 0.008$	$0.778 (\pm 0.022)$	$0.773 (\pm 0.056)$	$0.787 (\pm 0.008)$	$0.786 (\pm 0.009)$
DDC	$0.766 (\pm 0.012)$	$0.779 (\pm {0.010}^{\circ})$	0.780(±0.009	$0.779 (\pm 0.010)$	$0.778 (\pm 0.010)$	$0.767 (\pm 0.017)$	$0.768 (\pm 0.011)$	$0.780 (\pm 0.010)$	$0.770 (\pm 0.013)$
CMD	$0.767 (\pm 0.012)$	$0.791 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.789 (\pm 0.010)$	$0.792 (\pm 0.010)$	$0.765 (\pm 0.015)$	$0.710 (\pm 0.015)$	0.794(±0.009	0.785(±0.009
MMDA	$0.767 (\pm 0.011)$	$0.787 (\pm 0.011)$	$0.785 (\pm 0.010)$	$0.785 \hat{i} \pm 0.010 \hat{j}$	$0.787 (\pm 0.012)$	$0.769 (\pm 0.011)$	$0.766 (\pm {0.010}^{\circ})$	$0.787 (\pm 0.011$	$0.782 (\pm 0.011)$
CoDATS	$0.766 (\pm 0.013)$	$0.795 (\pm 0.009)$	$0.793 (\pm 0.010)$	$0.794 (\pm 0.012)$	$0.799 (\pm 0.010)$	$0.779 (\pm 0.016)$	$0.773 (\pm 0.020)$	$0.796 (\pm 0.009)$	$0.791 (\pm 0.015)$
Deep-Coral	$0.766 (\pm 0.012)$	0.784(±0.009	$0.783 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.782 (\pm 0.009)$	$0.769 (\pm 0.016)$	$0.769 (\pm {0.037}^{\circ})$	$0.785 (\pm 0.009)$	$0.776 (\pm 0.013)$
CDAN	$0.767 (\pm 0.012)$	$0.788 (\pm 0.010$	$0.787 (\pm 0.009)$	$0.787 (\pm {0.010}^{3})$	$0.787 (\pm 0.011)$	$0.775 (\pm 0.011)$	$0.776 (\pm 0.014)$	$0.788 (\pm 0.010)$	$0.777 (\pm 0.011)$
DANN	$0.767 (\pm 0.012)$	$0.796 (\pm 0.010)$	$0.792 (\pm 0.010)$	$0.793 / \pm {0.010}^{\circ}$	0.8001	$0.776 (\pm 0.011)$	$0.778 (\pm 0.012)$	$0.797 (\pm 0.009)$	$0.798 (\pm 0.012)$
DSAN	$0.769 (\pm 0.009)$	$0.796 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.791 (\pm 0.010)$	$0.800 (\pm 0.010)$	$0.779 (\pm 0.012)$	$0.763 (\pm 0.017)$	$0.795 (\pm 0.009)$	$0.789 (\pm 0.012)$
Avg.	$0.767 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.010)$	$0.789 (\pm 0.010)$	$0.772 (\pm 0.014)$	$0.764 (\pm 0.019)$	$0.788 (\pm 0.009)$	$0.781 (\pm 0.012)$

方法	亚马逊评论
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.769 (\pm 0.009)$	$0.777 (\pm 0.010)$	$0.778 (\pm 0.010)$	0.778(±0.011)	$0.777 (\pm 0.010)$	$0.765 (\pm 0.011)$	$0.766 (\pm 0.011)$	$0.778 (\pm 0.010)$	$0.769 (\pm 0.012)$
AdvSKM	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009$	$0.779 (\pm 0.010)$	$0.779 (\pm 0.008)$	$0.778 (\pm 0.011)$	$0.769 (\pm 0.012)$	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009)$	$0.770 (\pm 0.012)$
DIRT	$0.764 (\pm 0.009)$	$0.786 (\pm 0.008)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.008)$	$0.800 (\pm 0.008$	$0.778 (\pm 0.022)$	$0.773 (\pm 0.056)$	$0.787 (\pm 0.008)$	$0.786 (\pm 0.009)$
DDC	$0.766 (\pm 0.012)$	$0.779 (\pm {0.010}^{\circ})$	0.780(±0.009	$0.779 (\pm 0.010)$	$0.778 (\pm 0.010)$	$0.767 (\pm 0.017)$	$0.768 (\pm 0.011)$	$0.780 (\pm 0.010)$	$0.770 (\pm 0.013)$
CMD	$0.767 (\pm 0.012)$	$0.791 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.789 (\pm 0.010)$	$0.792 (\pm 0.010)$	$0.765 (\pm 0.015)$	$0.710 (\pm 0.015)$	0.794(±0.009	0.785(±0.009
MMDA	$0.767 (\pm 0.011)$	$0.787 (\pm 0.011)$	$0.785 (\pm 0.010)$	$0.785 \hat{i} \pm 0.010 \hat{j}$	$0.787 (\pm 0.012)$	$0.769 (\pm 0.011)$	$0.766 (\pm {0.010}^{\circ})$	$0.787 (\pm 0.011$	$0.782 (\pm 0.011)$
CoDATS	$0.766 (\pm 0.013)$	$0.795 (\pm 0.009)$	$0.793 (\pm 0.010)$	$0.794 (\pm 0.012)$	$0.799 (\pm 0.010)$	$0.779 (\pm 0.016)$	$0.773 (\pm 0.020)$	$0.796 (\pm 0.009)$	$0.791 (\pm 0.015)$
Deep-Coral	$0.766 (\pm 0.012)$	0.784(±0.009	$0.783 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.782 (\pm 0.009)$	$0.769 (\pm 0.016)$	$0.769 (\pm {0.037}^{\circ})$	$0.785 (\pm 0.009)$	$0.776 (\pm 0.013)$
CDAN	$0.767 (\pm 0.012)$	$0.788 (\pm 0.010$	$0.787 (\pm 0.009)$	$0.787 (\pm {0.010}^{3})$	$0.787 (\pm 0.011)$	$0.775 (\pm 0.011)$	$0.776 (\pm 0.014)$	$0.788 (\pm 0.010)$	$0.777 (\pm 0.011)$
DANN	$0.767 (\pm 0.012)$	$0.796 (\pm 0.010)$	$0.792 (\pm 0.010)$	$0.793 / \pm {0.010}^{\circ}$	0.8001	$0.776 (\pm 0.011)$	$0.778 (\pm 0.012)$	$0.797 (\pm 0.009)$	$0.798 (\pm 0.012)$
DSAN	$0.769 (\pm 0.009)$	$0.796 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.791 (\pm 0.010)$	$0.800 (\pm 0.010)$	$0.779 (\pm 0.012)$	$0.763 (\pm 0.017)$	$0.795 (\pm 0.009)$	$0.789 (\pm 0.012)$
平均	$0.767 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.010)$	$0.789 (\pm 0.010)$	$0.772 (\pm 0.014)$	$0.764 (\pm 0.019)$	$0.788 (\pm 0.009)$	$0.781 (\pm 0.012)$

Method	Sleep-EDF
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.676 (\pm 0.036)$	$0.722 (\pm 0.017)$	$0.719 (\pm 0.023)$	$0.718 (\pm 0.021)$	$0.724 (\pm 0.032)$	$0.726 (\pm 0.046)$	$0.678 (\pm 0.035)$	$0.747 (\pm 0.025)$	$0.715 (\pm 0.047)$
AdvSKM	$0.665 (\pm 0.058)$	$0.708 (\pm 0.023)$	$0.712 (\pm 0.027)$	$0.712 (\pm 0.032)$	$0.718 (\pm 0.030$	$0.703 (\pm {0.069}^{\circ}$	$0.692 (\pm 0.038)$	$0.722 (\pm 0.025)$	$0.706 (\pm 0.054)$
DIRT	$0.656 (\pm 0.058)$	$0.743 (\pm 0.009)$	$0.745 (\pm 0.012)$	$0.748 (\pm 0.019$	$0.742 (\pm 0.031)$	$0.679 (\pm 0.038)$	$0.686 (\pm 0.066)$	$0.749 (\pm 0.010)$	$0.728 (\pm 0.037)$
DDC	$0.646 (\pm 0.035)$	$0.717 (\pm 0.029)$	$0.721 (\pm 0.037$	$0.712 (\pm 0.031)$	$0.695 (\pm 0.020)$	$0.694 (\pm {0.056}^{\circ}$	$0.666 (\pm 0.031$	$0.724 (\pm 0.012)$	$0.704 (\pm 0.031)$
CMD	$0.653 (\pm 0.057)$	0.740(±0.022	$0.736 (\pm 0.016)$	$0.723 (\pm 0.020)$	$0.709 (\pm 0.015)$	$0.716 (\pm 0.052)$	$0.640 (\pm 0.068)$	$0.729 (\pm 0.018$	$0.725 (\pm 0.053)$
MMDA	$0.650 (\pm 0.051)$	$0.736 (\pm 0.014$	$0.727 (\pm 0.021)$	$0.723 (\pm 0.018)$	0.714(±0.028’	$0.704 (\pm 0.033)$	$0.660 (\pm 0.034)$	$0.745 (\pm 0.031)$	$0.715 (\pm 0.042)$
CoDATS	$0.672 (\pm 0.084)$	$0.738 (\pm 0.029)$	$0.739 (\pm 0.036$	$0.736 (\pm 0.030)$	$0.723 (\pm 0.039)$	$0.683 (\pm 0.090)$	$0.690 (\pm 0.107)$	0.744(±0.021	$0.715 (\pm 0.045)$
Deep-Coral	$0.643 (\pm 0.049)$	$0.716 (\pm 0.018)$	$0.717 (\pm 0.028$	$0.712 (\pm 0.027)$	$0.694 (\pm 0.032)$	$0.700 (\pm 0.053)$	$0.675 (\pm 0.077)$	$0.713 (\pm 0.021)$	$0.702 (\pm {0.070}^{\circ})$
CDAN	$0.652 (\pm 0.056)$	$0.732 (\pm 0.016)$	$0.739 (\pm 0.024)$	$0.739 (\pm 0.018)$	$0.728 (\pm 0.029)$	$0.697 (\pm 0.031)$	$0.642 (\pm 0.065)$	$0.748 (\pm 0.019$	$0.713 (\pm 0.045)$
DANN	$0.641 (\pm 0.047)$	$0.722 (\pm 0.017)$	0.723(±0.026	$0.721 (\pm 0.025)$	$0.714 (\pm 0.024)$	$0.687 (\pm 0.034)$	0.644(±0.046’	$0.724 (\pm 0.018)$	0.710(±0.035
DSAN	$0.653 (\pm 0.060)$	$0.748 (\pm 0.008)$	$0.740 (\pm 0.016)$	$0.732 (\pm 0.016)$	$0.728 (\pm 0.026)$	$0.712 (\pm {0.070}^{\circ})$	$0.589 (\pm 0.063)$	$0.757 (\pm 0.016)$	$0.700 (\pm 0.033)$
Avg.	$0.655 (\pm 0.054)$	$0.729 (\pm 0.018)$	$0.729 (\pm 0.024)$	$0.725 (\pm 0.023)$	$0.717 (\pm 0.028)$	$0.700 (\pm 0.052)$	$0.660 (\pm 0.057)$	$0.737 (\pm 0.020)$	$0.712 (\pm 0.045)$

方法	Sleep-EDF
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
HoMM	$0.676 (\pm 0.036)$	$0.722 (\pm 0.017)$	$0.719 (\pm 0.023)$	$0.718 (\pm 0.021)$	$0.724 (\pm 0.032)$	$0.726 (\pm 0.046)$	$0.678 (\pm 0.035)$	$0.747 (\pm 0.025)$	$0.715 (\pm 0.047)$
AdvSKM	$0.665 (\pm 0.058)$	$0.708 (\pm 0.023)$	$0.712 (\pm 0.027)$	$0.712 (\pm 0.032)$	$0.718 (\pm 0.030$	$0.703 (\pm {0.069}^{\circ}$	$0.692 (\pm 0.038)$	$0.722 (\pm 0.025)$	$0.706 (\pm 0.054)$
DIRT	$0.656 (\pm 0.058)$	$0.743 (\pm 0.009)$	$0.745 (\pm 0.012)$	$0.748 (\pm 0.019$	$0.742 (\pm 0.031)$	$0.679 (\pm 0.038)$	$0.686 (\pm 0.066)$	$0.749 (\pm 0.010)$	$0.728 (\pm 0.037)$
DDC	$0.646 (\pm 0.035)$	$0.717 (\pm 0.029)$	$0.721 (\pm 0.037$	$0.712 (\pm 0.031)$	$0.695 (\pm 0.020)$	$0.694 (\pm {0.056}^{\circ}$	$0.666 (\pm 0.031$	$0.724 (\pm 0.012)$	$0.704 (\pm 0.031)$
CMD	$0.653 (\pm 0.057)$	0.740(±0.022	$0.736 (\pm 0.016)$	$0.723 (\pm 0.020)$	$0.709 (\pm 0.015)$	$0.716 (\pm 0.052)$	$0.640 (\pm 0.068)$	$0.729 (\pm 0.018$	$0.725 (\pm 0.053)$
MMDA	$0.650 (\pm 0.051)$	$0.736 (\pm 0.014$	$0.727 (\pm 0.021)$	$0.723 (\pm 0.018)$	0.714(±0.028’	$0.704 (\pm 0.033)$	$0.660 (\pm 0.034)$	$0.745 (\pm 0.031)$	$0.715 (\pm 0.042)$
CoDATS	$0.672 (\pm 0.084)$	$0.738 (\pm 0.029)$	$0.739 (\pm 0.036$	$0.736 (\pm 0.030)$	$0.723 (\pm 0.039)$	$0.683 (\pm 0.090)$	$0.690 (\pm 0.107)$	0.744(±0.021	$0.715 (\pm 0.045)$
Deep-Coral	$0.643 (\pm 0.049)$	$0.716 (\pm 0.018)$	$0.717 (\pm 0.028$	$0.712 (\pm 0.027)$	$0.694 (\pm 0.032)$	$0.700 (\pm 0.053)$	$0.675 (\pm 0.077)$	$0.713 (\pm 0.021)$	$0.702 (\pm {0.070}^{\circ})$
CDAN	$0.652 (\pm 0.056)$	$0.732 (\pm 0.016)$	$0.739 (\pm 0.024)$	$0.739 (\pm 0.018)$	$0.728 (\pm 0.029)$	$0.697 (\pm 0.031)$	$0.642 (\pm 0.065)$	$0.748 (\pm 0.019$	$0.713 (\pm 0.045)$
DANN	$0.641 (\pm 0.047)$	$0.722 (\pm 0.017)$	0.723(±0.026	$0.721 (\pm 0.025)$	$0.714 (\pm 0.024)$	$0.687 (\pm 0.034)$	0.644(±0.046’	$0.724 (\pm 0.018)$	0.710(±0.035
DSAN	$0.653 (\pm 0.060)$	$0.748 (\pm 0.008)$	$0.740 (\pm 0.016)$	$0.732 (\pm 0.016)$	$0.728 (\pm 0.026)$	$0.712 (\pm {0.070}^{\circ})$	$0.589 (\pm 0.063)$	$0.757 (\pm 0.016)$	$0.700 (\pm 0.033)$
平均值	$0.655 (\pm 0.054)$	$0.729 (\pm 0.018)$	$0.729 (\pm 0.024)$	$0.725 (\pm 0.023)$	$0.717 (\pm 0.028)$	$0.700 (\pm 0.052)$	$0.660 (\pm 0.057)$	$0.737 (\pm 0.020)$	$0.712 (\pm 0.045)$

Method	UCI-HAR
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.782 (\pm 0.078)$	$0.833 (\pm 0.020)$	$0.818 (\pm 0.023)$	$0.818 (\pm 0.022)$	$0.783 (\pm 0.040)$	$0.809 (\pm 0.095)$	$0.800 (\pm 0.098)$	$0.826 (\pm 0.010)$	$0.854 (\pm 0.039)$
AdvSKM	$0.724 (\pm 0.059)$	$0.791 (\pm 0.024)$	$0.800 (\pm 0.022)$	$0.810 (\pm 0.022$	$0.768 (\pm 0.042)$	$0.707 (\pm 0.100)$	$0.711 (\pm 0.167)$	$0.800 (\pm 0.022)$	$0.811 (\pm 0.039)$
DIRT	$0.783 (\pm 0.044)$	$0.912 (\pm 0.013)$	$0.907 (\pm 0.009)$	$0.890 (\pm 0.016)$	$0.756 (\pm 0.036)$	$0.807 (\pm 0.107)$	$0.808 (\pm 0.112)$	$0.900 (\pm 0.015$	$0.928 (\pm 0.034)$
DDC	$0.790 (\pm 0.061)$	$0.806 (\pm 0.019)$	$0.807 (\pm 0.026)$	0.810(±0.017	$0.756 (\pm 0.108)$	$0.724 (\pm 0.066)$	$0.734 (\pm 0.109)$	$0.804 (\pm 0.028)$	$0.792 (\pm 0.013)$
CMD	$0.788 (\pm 0.058)$	$0.869 (\pm 0.012)$	$0.849 (\pm 0.014)$	$0.839 (\pm 0.023)$	$0.731 (\pm 0.066)$	$0.804 (\pm 0.064)$	$0.812 (\pm 0.080)$	$0.842 (\pm 0.025$	$0.888 (\pm 0.037)$
MMDA	$0.785 (\pm 0.018)$	$0.819 (\pm 0.022)$	$0.812 (\pm 0.028)$	$0.800 (\pm 0.032)$	$0.759 (\pm 0.085)$	$0.773 (\pm 0.073)$	$0.767 (\pm 0.107)$	$0.807 (\pm 0.025)$	$0.840 (\pm 0.055)$
CoDATS	$0.760 (\pm 0.037)$	$0.854 (\pm 0.022)$	$0.832 (\pm 0.027)$	$0.832 (\pm 0.006)$	$0.785 (\pm 0.057)$	$0.801 (\pm 0.079)$	$0.794 (\pm 0.078)$	$0.846 (\pm 0.016)$	$0.867 (\pm 0.012)$
Deep-Coral	$0.790 (\pm 0.035)$	$0.810 (\pm 0.007$	$0.800 (\pm 0.022)$	$0.808 (\pm 0.030)$	$0.771 (\pm 0.023)$	$0.768 (\pm 0.044)$	$0.773 (\pm 0.087)$	$0.808 (\pm 0.016)$	$0.806 (\pm 0.022)$
CDAN	$0.756 (\pm 0.055)$	$0.842 (\pm 0.009)$	$0.843 (\pm 0.020)$	$0.840 (\pm 0.034)$	$0.802 (\pm 0.080)$	$0.781 (\pm 0.072)$	$0.687 (\pm 0.068)$	$0.846 (\pm 0.018)$	$0.853 (\pm 0.026)$
DANN	$0.756 (\pm 0.026)$	$0.858 (\pm 0.016$	$0.856 (\pm 0.033)$	$0.856 (\pm 0.033)$	$0.800 (\pm {0.057}^{\circ})$	$0.763 (\pm 0.032)$	$0.780 (\pm 0.043)$	$0.849 (\pm 0.023)$	$0.847 (\pm 0.007)$
DSAN	$0.762 (\pm 0.032)$	$0.849 (\pm 0.023)$	$0.843 (\pm 0.033)$	$0.854 (\pm 0.025$	$0.749 (\pm 0.065)$	$0.775 (\pm 0.043)$	$0.744 (\pm 0.035)$	$0.858 (\pm 0.023)$	$0.865 (\pm 0.038)$
Avg.	$0.770 (\pm 0.046)$	$0.840 (\pm 0.017)$	$0.833 (\pm 0.023)$	$0.832 (\pm 0.024)$	$0.769 (\pm 0.060)$	$0.774 (\pm 0.070)$	$0.765 (\pm 0.090)$	$0.835 (\pm 0.020)$	$0.850 (\pm 0.029)$

方法	UCI-HAR
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.782 (\pm 0.078)$	$0.833 (\pm 0.020)$	$0.818 (\pm 0.023)$	$0.818 (\pm 0.022)$	$0.783 (\pm 0.040)$	$0.809 (\pm 0.095)$	$0.800 (\pm 0.098)$	$0.826 (\pm 0.010)$	$0.854 (\pm 0.039)$
AdvSKM	$0.724 (\pm 0.059)$	$0.791 (\pm 0.024)$	$0.800 (\pm 0.022)$	$0.810 (\pm 0.022$	$0.768 (\pm 0.042)$	$0.707 (\pm 0.100)$	$0.711 (\pm 0.167)$	$0.800 (\pm 0.022)$	$0.811 (\pm 0.039)$
DIRT	$0.783 (\pm 0.044)$	$0.912 (\pm 0.013)$	$0.907 (\pm 0.009)$	$0.890 (\pm 0.016)$	$0.756 (\pm 0.036)$	$0.807 (\pm 0.107)$	$0.808 (\pm 0.112)$	$0.900 (\pm 0.015$	$0.928 (\pm 0.034)$
DDC	$0.790 (\pm 0.061)$	$0.806 (\pm 0.019)$	$0.807 (\pm 0.026)$	0.810(±0.017	$0.756 (\pm 0.108)$	$0.724 (\pm 0.066)$	$0.734 (\pm 0.109)$	$0.804 (\pm 0.028)$	$0.792 (\pm 0.013)$
CMD	$0.788 (\pm 0.058)$	$0.869 (\pm 0.012)$	$0.849 (\pm 0.014)$	$0.839 (\pm 0.023)$	$0.731 (\pm 0.066)$	$0.804 (\pm 0.064)$	$0.812 (\pm 0.080)$	$0.842 (\pm 0.025$	$0.888 (\pm 0.037)$
MMDA	$0.785 (\pm 0.018)$	$0.819 (\pm 0.022)$	$0.812 (\pm 0.028)$	$0.800 (\pm 0.032)$	$0.759 (\pm 0.085)$	$0.773 (\pm 0.073)$	$0.767 (\pm 0.107)$	$0.807 (\pm 0.025)$	$0.840 (\pm 0.055)$
CoDATS	$0.760 (\pm 0.037)$	$0.854 (\pm 0.022)$	$0.832 (\pm 0.027)$	$0.832 (\pm 0.006)$	$0.785 (\pm 0.057)$	$0.801 (\pm 0.079)$	$0.794 (\pm 0.078)$	$0.846 (\pm 0.016)$	$0.867 (\pm 0.012)$
Deep-Coral	$0.790 (\pm 0.035)$	$0.810 (\pm 0.007$	$0.800 (\pm 0.022)$	$0.808 (\pm 0.030)$	$0.771 (\pm 0.023)$	$0.768 (\pm 0.044)$	$0.773 (\pm 0.087)$	$0.808 (\pm 0.016)$	$0.806 (\pm 0.022)$
CDAN	$0.756 (\pm 0.055)$	$0.842 (\pm 0.009)$	$0.843 (\pm 0.020)$	$0.840 (\pm 0.034)$	$0.802 (\pm 0.080)$	$0.781 (\pm 0.072)$	$0.687 (\pm 0.068)$	$0.846 (\pm 0.018)$	$0.853 (\pm 0.026)$
DANN	$0.756 (\pm 0.026)$	$0.858 (\pm 0.016$	$0.856 (\pm 0.033)$	$0.856 (\pm 0.033)$	$0.800 (\pm {0.057}^{\circ})$	$0.763 (\pm 0.032)$	$0.780 (\pm 0.043)$	$0.849 (\pm 0.023)$	$0.847 (\pm 0.007)$
DSAN	$0.762 (\pm 0.032)$	$0.849 (\pm 0.023)$	$0.843 (\pm 0.033)$	$0.854 (\pm 0.025$	$0.749 (\pm 0.065)$	$0.775 (\pm 0.043)$	$0.744 (\pm 0.035)$	$0.858 (\pm 0.023)$	$0.865 (\pm 0.038)$
平均	$0.770 (\pm 0.046)$	$0.840 (\pm 0.017)$	$0.833 (\pm 0.023)$	$0.832 (\pm 0.024)$	$0.769 (\pm 0.060)$	$0.774 (\pm 0.070)$	$0.765 (\pm 0.090)$	$0.835 (\pm 0.020)$	$0.850 (\pm 0.029)$

Method	HHAR
		Heuristic				│Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.739 (\pm 0.044)$	$0.757 (\pm 0.014)$	0.759(±0.013)	$0.759 (\pm 0.011)$	$0.700 (\pm 0.058)$	$0.720 (\pm 0.027)$	$0.733 (\pm 0.031)$	$0.759 (\pm 0.007)$	$0.764 (\pm 0.023)$
AdvSKM	$0.718 (\pm 0.042)$	$0.749 (\pm 0.027)$	$0.742 (\pm 0.032)$	0.748(±0.034)	$0.676 (\pm 0.046)$	$0.730 (\pm 0.051)$	$0.728 (\pm 0.051)$	$0.752 (\pm 0.031)$	$0.749 (\pm 0.025)$
DIRT	$0.728 (\pm 0.026)$	$0.803 (\pm 0.011$	$0.792 (\pm 0.016)$	$0.803 (\pm 0.017)$	$0.796 (\pm 0.066)$	$0.743 (\pm 0.028)$	$0.739 (\pm 0.075)$	$0.816 (\pm 0.008)$	$0.820 (\pm 0.015)$
DDC	$0.716 (\pm 0.063)$	$0.748 (\pm 0.014)$	$0.750 (\pm 0.009$	$0.748 (\pm 0.007)$	$0.717 (\pm 0.075)$	$0.711 (\pm 0.048)$	$0.705 (\pm 0.066)$	$0.748 (\pm 0.012)$	$0.729 (\pm 0.027)$
CMD	$0.748 (\pm 0.027)$	$0.760 (\pm 0.014)$	$0.764 (\pm 0.006)$	0.767(±0.007	$0.737 (\pm 0.100)$	$0.775 (\pm 0.031$	$0.643 (\pm 0.031)$	$0.766 (\pm 0.016)$	$0.794 (\pm 0.023)$
MMDA	$0.738 (\pm 0.036)$	0.783(±0.017	$0.781 (\pm 0.016)$	$0.780 (\pm 0.015)$	$0.698 (\pm 0.038)$	$0.719 (\pm 0.036)$	$0.731 (\pm 0.047)$	$0.780 (\pm 0.017)$	$0.785 (\pm 0.035)$
CoDATS	$0.710 (\pm 0.030)$	$0.766 (\pm 0.023)$	$0.772 (\pm 0.040)$	0.773(±0.050	$0.722 (\pm 0.064)$	$0.739 (\pm 0.028)$	$0.739 (\pm 0.040)$	$0.812 (\pm 0.009)$	$0.785 \hat{i} \pm 0.039$
Deep-Coral	$0.745 (\pm 0.046)$	0.766(+0.012	$0.762 (\pm 0.015)$	$0.766 (\pm 0.027)$	$0.681 (\pm 0.073)$	$0.754 (\pm {0.054}^{\circ})$	$0.758 (\pm 0.244)$	$0.764 (\pm 0.006)$	$0.776 (\pm 0.023)$
CDAN	$0.728 (\pm 0.039)$	$0.762 (\pm 0.012)$	$0.758 (\pm 0.017)$	$0.764 (\pm 0.016)$	0.765(±0.063)	$0.774 (\pm 0.035)$	$0.775 (\pm 0.036)$	0.816(±0.011	$0.790 (\pm 0.038)$
DANN	$0.757 (\pm 0.057)$	0.779(±0.012)	$0.774 (\pm {0.009}^{3})$	$0.773 (\pm 0.011)$	$0.722 (\pm 0.103)$	$0.798 (\pm 0.041)$	$0.793 (\pm 0.045)$	0.818(±0.009	$0.807 (\pm 0.020)$
DSAN	$0.721 (\pm 0.053)$	0.803(±0.010)	$0.797 (\pm 0.014)$	$0.802 (\pm 0.007)$	$0.724 (\pm 0.065)$	$0.741 (\pm 0.033)$	$0.596 (\pm 0.031)$	$0.825 (\pm 0.008)$	$0.826 \hat{i} \pm 0.046 \hat{j}$
Avg.	$0.732 (\pm 0.042)$	$0.771 (\pm 0.015)$	$0.768 (\pm 0.017)$	0.771(±0.018)	$0.722 (\pm 0.068)$	$0.746 (\pm 0.037)$	$0.722 (\pm 0.063)$	$0.787 (\pm 0.012)$	$0.784 (\pm 0.028)$

方法	HHAR
		启发式				│理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.739 (\pm 0.044)$	$0.757 (\pm 0.014)$	0.759(±0.013)	$0.759 (\pm 0.011)$	$0.700 (\pm 0.058)$	$0.720 (\pm 0.027)$	$0.733 (\pm 0.031)$	$0.759 (\pm 0.007)$	$0.764 (\pm 0.023)$
AdvSKM	$0.718 (\pm 0.042)$	$0.749 (\pm 0.027)$	$0.742 (\pm 0.032)$	0.748(±0.034)	$0.676 (\pm 0.046)$	$0.730 (\pm 0.051)$	$0.728 (\pm 0.051)$	$0.752 (\pm 0.031)$	$0.749 (\pm 0.025)$
DIRT	$0.728 (\pm 0.026)$	$0.803 (\pm 0.011$	$0.792 (\pm 0.016)$	$0.803 (\pm 0.017)$	$0.796 (\pm 0.066)$	$0.743 (\pm 0.028)$	$0.739 (\pm 0.075)$	$0.816 (\pm 0.008)$	$0.820 (\pm 0.015)$
DDC	$0.716 (\pm 0.063)$	$0.748 (\pm 0.014)$	$0.750 (\pm 0.009$	$0.748 (\pm 0.007)$	$0.717 (\pm 0.075)$	$0.711 (\pm 0.048)$	$0.705 (\pm 0.066)$	$0.748 (\pm 0.012)$	$0.729 (\pm 0.027)$
CMD	$0.748 (\pm 0.027)$	$0.760 (\pm 0.014)$	$0.764 (\pm 0.006)$	0.767(±0.007	$0.737 (\pm 0.100)$	$0.775 (\pm 0.031$	$0.643 (\pm 0.031)$	$0.766 (\pm 0.016)$	$0.794 (\pm 0.023)$
MMDA	$0.738 (\pm 0.036)$	0.783(±0.017	$0.781 (\pm 0.016)$	$0.780 (\pm 0.015)$	$0.698 (\pm 0.038)$	$0.719 (\pm 0.036)$	$0.731 (\pm 0.047)$	$0.780 (\pm 0.017)$	$0.785 (\pm 0.035)$
CoDATS	$0.710 (\pm 0.030)$	$0.766 (\pm 0.023)$	$0.772 (\pm 0.040)$	0.773(±0.050	$0.722 (\pm 0.064)$	$0.739 (\pm 0.028)$	$0.739 (\pm 0.040)$	$0.812 (\pm 0.009)$	$0.785 \hat{i} \pm 0.039$
Deep-Coral	$0.745 (\pm 0.046)$	0.766(+0.012	$0.762 (\pm 0.015)$	$0.766 (\pm 0.027)$	$0.681 (\pm 0.073)$	$0.754 (\pm {0.054}^{\circ})$	$0.758 (\pm 0.244)$	$0.764 (\pm 0.006)$	$0.776 (\pm 0.023)$
CDAN	$0.728 (\pm 0.039)$	$0.762 (\pm 0.012)$	$0.758 (\pm 0.017)$	$0.764 (\pm 0.016)$	0.765(±0.063)	$0.774 (\pm 0.035)$	$0.775 (\pm 0.036)$	0.816(±0.011	$0.790 (\pm 0.038)$
DANN	$0.757 (\pm 0.057)$	0.779(±0.012)	$0.774 (\pm {0.009}^{3})$	$0.773 (\pm 0.011)$	$0.722 (\pm 0.103)$	$0.798 (\pm 0.041)$	$0.793 (\pm 0.045)$	0.818(±0.009)	$0.807 (\pm 0.020)$
DSAN	$0.721 (\pm 0.053)$	0.803(±0.010)	$0.797 (\pm 0.014)$	$0.802 (\pm 0.007)$	$0.724 (\pm 0.065)$	$0.741 (\pm 0.033)$	$0.596 (\pm 0.031)$	$0.825 (\pm 0.008)$	$0.826 \hat{i} \pm 0.046 \hat{j}$
平均	$0.732 (\pm 0.042)$	$0.771 (\pm 0.015)$	$0.768 (\pm 0.017)$	0.771(±0.018)	$0.722 (\pm 0.068)$	$0.746 (\pm 0.037)$	$0.722 (\pm 0.063)$	$0.787 (\pm 0.012)$	$0.784 (\pm 0.028)$

Method	WISDM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.753 (\pm 0.054)$	$0.741 (\pm 0.026)$	$0.738 (\pm 0.031)$	$0.739 (\pm 0.047)$	$0.775 (\pm 0.062)$	$0.753 (\pm 0.054)$	$0.740 (\pm 0.054)$	$0.728 (\pm 0.021)$	$0.774 (\pm 0.037)$
AdvSKM	$0.747 (\pm 0.050)$	$0.771 (\pm 0.043)$	$0.781 (\pm 0.055)$	$0.779 (\pm 0.035)$	$0.742 (\pm 0.062)$	$0.747 (\pm 0.050)$	$0.747 (\pm 0.135)$	$0.777 (\pm 0.031)$	$0.779 (\pm 0.041)$
DIRT	$0.738 (\pm 0.038)$	$0.792 (\pm 0.015)$	0.797(±0.024	$0.797 (\pm 0.037)$	$0.756 (\pm 0.071)$	$0.738 (\pm 0.059)$	$0.797 (\pm 0.059)$	$0.798 (\pm 0.018)$	$0.816 (\pm 0.063)$
DDC	$0.741 (\pm 0.071)$	$0.780 (\pm 0.032)$	$0.779 (\pm 0.052)$	$0.787 (\pm 0.049)$	$0.737 (\pm 0.071)$	$0.741 (\pm 0.076)$	0.741(±0.063	$0.782 (\pm 0.038)$	$0.770 (\pm 0.060)$
CMD	$0.710 (\pm 0.088)$	$0.772 (\pm 0.021$	$0.765 (\pm 0.032)$	$0.767 (\pm 0.040)$	$0.728 (\pm 0.092)$	$0.713 (\pm 0.084)$	$0.686 (\pm 0.113)$	$0.773 (\pm 0.032)$	$0.742 (\pm 0.071)$
MMDA	$0.759 (\pm 0.047)$	$0.789 (\pm 0.017)$	$0.772 (\pm 0.030)$	$0.745 (\pm 0.035)$	$0.754 (\pm 0.050)$	$0.759 (\pm 0.047)$	$0.750 (\pm 0.047)$	$0.790 (\pm 0.018)$	$0.775 (\pm 0.030)$
CoDATS	$0.711 (\pm 0.039)$	0.775(±0.018	$0.757 (\pm 0.027)$	$0.751 (\pm 0.020)$	$0.682 (\pm 0.057)$	$0.709 (\pm 0.039)$	$0.735 (\pm 0.054)$	$0.764 (\pm 0.015)$	$0.770 (\pm 0.019)$
Deep-Coral	$0.694 (\pm 0.030)$	$0.717 (\pm 0.041)$	0.723(±0.037	$0.713 (\pm 0.035)$	$0.664 (\pm 0.055)$	$0.694 (\pm 0.030)$	$0.670 (\pm 0.149)$	$0.723 (\pm 0.026)$	$0.736 (\pm 0.044)$
CDAN	$0.760 (\pm 0.057)$	$0.762 (\pm 0.048)$	$0.762 (\pm 0.046)$	0.781(±0.051	$0.768 (\pm 0.074)$	$0.760 (\pm 0.057)$	$0.750 (\pm 0.091)$	$0.765 (\pm 0.040)$	$0.779 (\pm 0.049)$
DANN	$0.724 (\pm 0.042)$	$0.789 (\pm 0.018)$	$0.802 (\pm 0.028)$	$0.796 (\pm 0.036)$	$0.745 (\pm 0.026)$	$0.720 (\pm 0.042)$	$0.702 (\pm 0.053)$	$0.778 (\pm 0.019)$	$0.765 (\pm 0.044)$
DSAN	$0.759 (\pm 0.030)$	$0.765 (\pm 0.024)$	$0.769 (\pm 0.034)$	$0.756 (\pm 0.025)$	$0.757 (\pm 0.068)$	$0.759 (\pm 0.030)$	$0.663 (\pm 0.025)$	$0.722 (\pm 0.013)$	$0.779 (\pm 0.044)$
Avg.	$0.736 (\pm 0.050)$	$0.768 (\pm 0.027)$	$0.768 (\pm 0.036)$	$0.765 (\pm 0.037)$	$0.737 (\pm 0.062)$	$0.736 (\pm 0.052)$	$0.726 (\pm 0.077)$	$0.764 (\pm 0.025)$	$0.771 (\pm 0.046)$

方法	WISDM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.753 (\pm 0.054)$	$0.741 (\pm 0.026)$	$0.738 (\pm 0.031)$	$0.739 (\pm 0.047)$	$0.775 (\pm 0.062)$	$0.753 (\pm 0.054)$	$0.740 (\pm 0.054)$	$0.728 (\pm 0.021)$	$0.774 (\pm 0.037)$
AdvSKM	$0.747 (\pm 0.050)$	$0.771 (\pm 0.043)$	$0.781 (\pm 0.055)$	$0.779 (\pm 0.035)$	$0.742 (\pm 0.062)$	$0.747 (\pm 0.050)$	$0.747 (\pm 0.135)$	$0.777 (\pm 0.031)$	$0.779 (\pm 0.041)$
DIRT	$0.738 (\pm 0.038)$	$0.792 (\pm 0.015)$	0.797(±0.024	$0.797 (\pm 0.037)$	$0.756 (\pm 0.071)$	$0.738 (\pm 0.059)$	$0.797 (\pm 0.059)$	$0.798 (\pm 0.018)$	$0.816 (\pm 0.063)$
DDC	$0.741 (\pm 0.071)$	$0.780 (\pm 0.032)$	$0.779 (\pm 0.052)$	$0.787 (\pm 0.049)$	$0.737 (\pm 0.071)$	$0.741 (\pm 0.076)$	0.741(±0.063	$0.782 (\pm 0.038)$	$0.770 (\pm 0.060)$
CMD	$0.710 (\pm 0.088)$	$0.772 (\pm 0.021$	$0.765 (\pm 0.032)$	$0.767 (\pm 0.040)$	$0.728 (\pm 0.092)$	$0.713 (\pm 0.084)$	$0.686 (\pm 0.113)$	$0.773 (\pm 0.032)$	$0.742 (\pm 0.071)$
MMDA	$0.759 (\pm 0.047)$	$0.789 (\pm 0.017)$	$0.772 (\pm 0.030)$	$0.745 (\pm 0.035)$	$0.754 (\pm 0.050)$	$0.759 (\pm 0.047)$	$0.750 (\pm 0.047)$	$0.790 (\pm 0.018)$	$0.775 (\pm 0.030)$
CoDATS	$0.711 (\pm 0.039)$	0.775(±0.018	$0.757 (\pm 0.027)$	$0.751 (\pm 0.020)$	$0.682 (\pm 0.057)$	$0.709 (\pm 0.039)$	$0.735 (\pm 0.054)$	$0.764 (\pm 0.015)$	$0.770 (\pm 0.019)$
Deep-Coral	$0.694 (\pm 0.030)$	$0.717 (\pm 0.041)$	0.723(±0.037	$0.713 (\pm 0.035)$	$0.664 (\pm 0.055)$	$0.694 (\pm 0.030)$	$0.670 (\pm 0.149)$	$0.723 (\pm 0.026)$	$0.736 (\pm 0.044)$
CDAN	$0.760 (\pm 0.057)$	$0.762 (\pm 0.048)$	$0.762 (\pm 0.046)$	0.781(±0.051	$0.768 (\pm 0.074)$	$0.760 (\pm 0.057)$	$0.750 (\pm 0.091)$	$0.765 (\pm 0.040)$	$0.779 (\pm 0.049)$
DANN	$0.724 (\pm 0.042)$	$0.789 (\pm 0.018)$	$0.802 (\pm 0.028)$	$0.796 (\pm 0.036)$	$0.745 (\pm 0.026)$	$0.720 (\pm 0.042)$	$0.702 (\pm 0.053)$	$0.778 (\pm 0.019)$	$0.765 (\pm 0.044)$
DSAN	$0.759 (\pm 0.030)$	$0.765 (\pm 0.024)$	$0.769 (\pm 0.034)$	$0.756 (\pm 0.025)$	$0.757 (\pm 0.068)$	$0.759 (\pm 0.030)$	$0.663 (\pm 0.025)$	$0.722 (\pm 0.013)$	$0.779 (\pm 0.044)$
平均值	$0.736 (\pm 0.050)$	$0.768 (\pm 0.027)$	$0.768 (\pm 0.036)$	$0.765 (\pm 0.037)$	$0.737 (\pm 0.062)$	$0.736 (\pm 0.052)$	$0.726 (\pm 0.077)$	$0.764 (\pm 0.025)$	$0.771 (\pm 0.046)$

A. Stisen, H. Blunck, S. Bhattacharya, T. S. Prentow, M. B. Kjærgaard, A. Dey, T. Sonne, and M. M. Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, SenSys '15, pp. 127-140, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336314. doi: 10.1145/2809695.2809718.

A. Stisen, H. Blunck, S. Bhattacharya, T. S. Prentow, M. B. Kjærgaard, A. Dey, T. Sonne, 和 M. M. Jensen. 智能设备是不同的：评估和缓解移动感知异质性以进行活动识别. 在第13届ACM嵌入式网络传感器系统会议论文集，SenSys '15, 第127-140页, 纽约, NY, 美国, 2015. 计算机协会. ISBN 9781450336314. doi: 10.1145/2809695.2809718.

Spaces: input space $X \subset R^{d_{1}}$ and label space $Y$ with inner product $⟨ .,, ⟩_{V} . Y$ is assumed to be a separable Hilbert space such that for the associated norm $∥ y ∥_{V} \leq y_{0}$ holds for all $y \in Y$ and some $y_{0} > 0$ . Note that this setting is more general than the one from the main text,where we assumed $Y \subset R^{d_{2}}$ (the simplification in the main text improves readability and respectation of space limits).

空间：输入空间 $X \subset R^{d_{1}}$ 和标签空间 $Y$ 的内积 $⟨ .,, ⟩_{V} . Y$ 假定为可分的希尔伯特空间，对于相关的范数 $∥ y ∥_{V} \leq y_{0}$ ，对所有 $y \in Y$ 和某些 $y_{0} > 0$ 都成立。请注意，这种设置比主文本中的设置更为一般，在主文本中我们假设了 $Y \subset R^{d_{2}}$ （主文本中的简化提高了可读性并尊重空间限制）。

Datasets and Distributions: Source data set: $(x, y) = ((x_{1}, y_{1}), \dots, (x_{n}, y_{n})) \in {(X \times Y)}^{n}$ independently drawn according to source distribution $p$ on $X \times Y$ and an unlabeled target dataset $x^{'} = (x_{1}^{'}, \dots, x_{m}^{'}) \in X^{m}$ independently drawn according marginal distribution $q_{X}$ similarly denoted as $p_{X}$ ).

数据集和分布：源数据集： $(x, y) = ((x_{1}, y_{1}), \dots, (x_{n}, y_{n})) \in {(X \times Y)}^{n}$ 根据源分布 $p$ 在 $X \times Y$ 上独立抽取，未标记的目标数据集 $x^{'} = (x_{1}^{'}, \dots, x_{m}^{'}) \in X^{m}$ 根据边际分布 $q_{X}$ 独立抽取，类似地表示为 $p_{X}$ 。

Notation from functional analysis/operator theory Let

U

and

V

denote separable Hilbert spaces (i.e. they admit countable orthonormal bases) with associated inner products

⟨ \cdot, \cdot ⟩_{U}

(or

⟨ \cdot, \cdot ⟩_{V}

, respectively). Let us briefly recall some notions from functional analysis that we need in order to set up our theory. There are lots of standard references on these aspects, e.g. Teschl (2022a) and Teschl (2022b):

从泛函分析/算子理论的符号设

U

和

V

表示可分的希尔伯特空间（即它们具有可数的正交基）及其相关的内积

⟨ \cdot, \cdot ⟩_{U}

（或

⟨ \cdot, \cdot ⟩_{V}

，分别）。我们简要回顾一些我们在建立理论时需要的泛函分析概念。关于这些方面有很多标准参考文献，例如 Teschl (2022a) 和 Teschl (2022b):

For (probability) measure $q$ on $X$ (or $Y$ ) and appropriate functions $F : X \to U$ (e.g. strongly measurable and $∥ F ∥_{U}$ is integrable wrt. $q$ ) we denote the usual (U-valued) Bochner integral of $F$ as $\int_{X} F (x) d q (x)$ . We denote the associated $L^{p}$ -spaces by $L^{p} (X, q, U)$ ,or $L^{p} (q)$ for short, if the associated spaces are clear from the context.

对于 $q$ 在 $X$ (或 $Y$ ) 上的（概率）测度和适当的函数 $F : X \to U$ (例如强可测且 $∥ F ∥_{U}$ 关于 $q$ 可积)，我们将 $F$ 的通常（U值）博赫纳积分表示为 $\int_{X} F (x) d q (x)$ 。如果相关空间从上下文中可以清楚地看出，我们将相关的 $L^{p}$ 空间表示为 $L^{p} (X, q, U)$ ，或简写为 $L^{p} (q)$ 。

Assumptions on models We assume that the regression function

f^{*} = f_{p} = f_{q}

as well as the models

f_{1}, \dots, f_{l}

belong to a hypothesis space

H \subseteq C (X, Y) \subseteq L^{2} (p_{X}) \cap L^{2} (q_{X})

,where

C (X, Y)

denotes the space of bounded continous functions

X \to Y

. The space

H

should satisfy the following assumptions, which are discussed in much greater detail in Caponnetto & De Vito (2007) and Caponnetto & De Vito (2005):

模型假设我们假设回归函数

f^{*} = f_{p} = f_{q}

以及模型

f_{1}, \dots, f_{l}

属于假设空间

H \subseteq C (X, Y) \subseteq L^{2} (p_{X}) \cap L^{2} (q_{X})

，其中

C (X, Y)

表示有界连续函数的空间

X \to Y

。空间

H

应满足以下假设，这些假设在 Caponnetto & De Vito (2007) 和 Caponnetto & De Vito (2005) 中有更详细的讨论：

Next we observe that the best approximation

f^{*}

of the target regression function

f_{q}

by linear combinations corresponds to the vector

c^{*} = (c_{1}^{*}, \dots, c_{l}^{*})

of ideal coefficients in (15) that solves the linear system

G c^{*} = \bar{g}

with the Gram matrix

G = {({⟨ \sqrt{T_{q_{X}}} f_{k}, \sqrt{T_{q_{X}}} f_{u} ⟩}_{H})}_{k, u = 1}^{l}

and the right-hand side vector

\bar{g} = {({⟨ \sqrt{T_{q_{X}}} f_{q}, \sqrt{T_{q_{X}}} f_{k} ⟩}_{H})}_{k = 1}^{l}

. Let us provide a prove of this short observation in the next lemma. Note that the entries

G

and

g

can equivalently also be formulated in terms of

⟨ \cdot, \cdot ⟩_{L^{2} (g x)}

, as done in the main text. We are going to use this formulation in the next lemma in order to be compatible with the main text (switching to the inner products in terms of

H

would not change the argument of the proof at all):

接下来我们观察到，目标回归函数

f_{q}

的最佳近似

f^{*}

通过线性组合对应于理想系数向量

c^{*} = (c_{1}^{*}, \dots, c_{l}^{*})

，该向量在 (15) 中解决了与 Gram 矩阵

G = {({⟨ \sqrt{T_{q_{X}}} f_{k}, \sqrt{T_{q_{X}}} f_{u} ⟩}_{H})}_{k, u = 1}^{l}

和右侧向量

\bar{g} = {({⟨ \sqrt{T_{q_{X}}} f_{q}, \sqrt{T_{q_{X}}} f_{k} ⟩}_{H})}_{k = 1}^{l}

的线性系统。我们将在下一个引理中提供这个简短观察的证明。注意，条目

G

和

g

也可以等效地用

⟨ \cdot, \cdot ⟩_{L^{2} (g x)}

表述，如主文本中所做。我们将在下一个引理中使用这种表述，以便与主文本兼容（切换到

H

的内积不会改变证明的论证）：

With the matrix

\tilde{G}

at hand one can easily check whether or not it is well-conditioned and

{\tilde{G}}^{- 1}

exists (otherwise one needs to get rid of models with similar performance). Thus the norms

∥ \tilde{G} ∥_{L (R^{l})}

and

{‖ \begin{matrix} {\tilde{G}}^{- 1} \end{matrix} ‖}_{L (R^{l})}

can be bounded independently of

m

and

n

,due to the fact that all their entries can be bounded as follows (we only do the calculation for the entries of

\tilde{G}

有了矩阵

\tilde{G}

，可以很容易地检查它是否条件良好以及

{\tilde{G}}^{- 1}

是否存在（否则需要剔除性能相似的模型）。因此，范数

∥ \tilde{G} ∥_{L (R^{l})}

和

{‖ \begin{matrix} {\tilde{G}}^{- 1} \end{matrix} ‖}_{L (R^{l})}

可以独立于

m

和

n

进行界定，因为它们的所有条目可以如下界定（我们仅对

\tilde{G}

的条目进行计算）：

On the dependence of the error bound on the number

l

of models An interesting question is, how the bound in Eq. (4) depends on

l

. To this end,let us have a look at the second term in the last line in Eq. (24):

{‖ \begin{matrix} \sqrt{T_{q_{X}}} \end{matrix} ‖}_{L}

analyzes a sampling operator,thus does not depend on the number of models,same goes for

γ_{l}

,which is just a uniform bound on all our models. To analyze

{‖ \begin{matrix} c^{*} - \tilde{c} \end{matrix} ‖}_{R^{2}}^{2}

, let us have a look at the individual factors in the first inequality of Eq. (23). To not overload notation,

C > 0

is used here for any absolute constant that is independent of

l, m, n

and

δ

关于误差界限对模型数量

l

的依赖，一个有趣的问题是，方程 (4) 中的界限如何依赖于

l

。为此，让我们看看方程 (24) 最后一行中的第二项：

{‖ \begin{matrix} \sqrt{T_{q_{X}}} \end{matrix} ‖}_{L}

分析了一个采样算子，因此不依赖于模型的数量，

γ_{l}

也是如此，它只是对我们所有模型的一个统一界限。为了分析

{‖ \begin{matrix} c^{*} - \tilde{c} \end{matrix} ‖}_{R^{2}}^{2}

，让我们看看方程 (23) 第一个不等式中的各个因子。为了不使符号过于复杂，这里使用

C > 0

表示任何与

l, m, n

和

δ

无关的绝对常数。

$∥ G - \tilde{G} ∥_{L (R^{l})}$ : The individual entries of $G - \tilde{G}$ are (in absolute values) bounded by Lemma 5. The proof arguments only involve norm bounds on the associated sampling operators, the uniform bound $γ_{l}$ on all the models and the bound $B$ on $β$ ,thus the absolute constant $C$ there is independent of $l$ . By the definition of the matrix Frobenius norm,we thus get

$∥ G - \tilde{G} ∥_{L (R^{l})}$ : $G - \tilde{G}$ 的各个条目（绝对值）由引理 5 限制。证明论证仅涉及与相关采样算子的范数界限、对所有模型的统一界限 $γ_{l}$ 和对 $β$ 的界限 $B$ ，因此那里绝对常数 $C$ 与 $l$ 无关。根据矩阵 Frobenius 范数的定义，我们得到

$∥ \tilde{g} - \bar{g} ∥_{R^{l}}$ : Similar arguments as before lead to $∥ \tilde{g} - \bar{g} ∥_{L (R^{l})} \leq C \sqrt{l} (\log^{\frac{1}{2}} \frac{1}{δ}) (n^{- \frac{1}{2}} + m^{- \frac{1}{2}})$ - ${‖ \begin{matrix} G^{- 1} \end{matrix} ‖}_{L (R^{l})}$ : It is natural to assume that there is some constant $c \geq {‖ \begin{matrix} {\tilde{G}}^{- 1} \end{matrix} ‖}_{L (R^{l})}$ . Otherwise, we can, e.g., orthogonalize our models and coefficients without changing the aggregation, but with reducing the conditioning number (i.e.,with reducing $‖ \begin{matrix} {\tilde{G}}^{- 1} \end{matrix} ‖$ ). It is also natural to assume that $m$ and $n$ are large enough such that $l (n^{- \frac{1}{2}} + m^{- \frac{1}{2}}) < \frac{1}{2 c}$ . Then applying Eq. (25) to Eq. (22),we can deduce that ${‖ \begin{matrix} G^{- 1} \end{matrix} ‖}_{L (R^{l})} \leq 2 c$ .

$∥ \tilde{g} - \bar{g} ∥_{R^{l}}$ : 与之前类似的论点导致 $∥ \tilde{g} - \bar{g} ∥_{L (R^{l})} \leq C \sqrt{l} (\log^{\frac{1}{2}} \frac{1}{δ}) (n^{- \frac{1}{2}} + m^{- \frac{1}{2}})$ - ${‖ \begin{matrix} G^{- 1} \end{matrix} ‖}_{L (R^{l})}$ : 假设存在某个常数 $c \geq {‖ \begin{matrix} {\tilde{G}}^{- 1} \end{matrix} ‖}_{L (R^{l})}$ 是自然的。否则，我们可以，例如，在不改变聚合的情况下正交化我们的模型和系数，但可以减少条件数（即，减少 $‖ \begin{matrix} {\tilde{G}}^{- 1} \end{matrix} ‖$ ）。同样，假设 $m$ 和 $n$ 足够大，以至于 $l (n^{- \frac{1}{2}} + m^{- \frac{1}{2}}) < \frac{1}{2 c}$ 也是自然的。然后将方程 (25) 应用到方程 (22)，我们可以推导出 ${‖ \begin{matrix} G^{- 1} \end{matrix} ‖}_{L (R^{l})} \leq 2 c$ 。

Let us give a short discussion on the construction of our required function space mentioned in the previous Section A,the reproducing kernel space

H

. As mentioned already in the main text,the explicit knowledge of

H

is not required,we just need to rely on its existence. First,any of our models

f

can be regarded as an element of some reproducing kernel space (RKHS)

\tilde{H}

satisfying the assumptions 1 . This is immediate if

f : X \to R

is a real valued continuous function and we take

k (x, y) = f (x) f (y)

as the associated reproducing kernel. In the case

f : X \to Y

and

Y

is finite dimensional, it is not hard to see that a similar construction is possible, as this case can again be boiled down to the construction of a kernel with real-valued output, see e.g. Remark 1 in Caponnetto & De Vito (2007) for details.

让我们简要讨论一下在前一节 A 中提到的所需函数空间的构造，即再生核空间

H

。如主文本中所提到的，不需要明确知道

H

，我们只需依赖于其存在性。首先，我们的任何模型

f

都可以被视为某个满足假设 1 的再生核希尔伯特空间 (RKHS)

\tilde{H}

的元素。如果

f : X \to R

是一个实值连续函数，并且我们将

k (x, y) = f (x) f (y)

作为相关的再生核，这一点是显而易见的。在

f : X \to Y

和

Y

是有限维的情况下，不难看出类似的构造是可能的，因为这种情况可以再次归结为构造一个具有实值输出的核，具体细节请参见 Caponnetto & De Vito (2007) 中的备注 1。

Overall we end up with a finite sequence of spaces

{(H_{k})}_{k = 1}^{l + 1}

of functions living on the same domain

X

(we have

l + 1

as we also take into account the regression function),and the existence of a RKHS containing all given models and the regression function is not a real restriction. For example, in case of real valued functions, this assumption is automatically satisfied, as linear combinations of functions with the same domain which stem from a finite sequence of RKHSs belong to an RKHS. This follows from a classical result by N. Aronszajn and R. Godement, see e.g. Pereverzyev (2022, Theorem 1.4.).

总体而言，我们最终得到一个有限的空间序列

{(H_{k})}_{k = 1}^{l + 1}

，这些函数生活在同一领域

X

（我们有

l + 1

，因为我们还考虑了回归函数），并且包含所有给定模型和回归函数的 RKHS 的存在并不是一个真正的限制。例如，在实值函数的情况下，这个假设是自动满足的，因为来自有限序列的 RKHS 的同一领域的函数的线性组合属于一个 RKHS。这是 N. Aronszajn 和 R. Godement 的经典结果，参见例如 Pereverzyev (2022, Theorem 1.4.)。

There is also ongoing research on constructing function spaces (and especially associated reproducing ReLU networks, Fermanian et al. (2021) for recurrent networks and Bietti & Mairal (2017; 2019) for convolutional neural networks. Incorporating these into our work may lead to refined generalization bounds that also reflect the nature of our models. We leave the details open for future work.

目前也在进行关于构建函数空间的研究（特别是与之相关的再生 ReLU 网络，Fermanian 等人 (2021) 针对递归网络，Bietti & Mairal (2017; 2019) 针对卷积神经网络）。将这些纳入我们的工作可能会导致更精细的泛化界限，这些界限也反映了我们模型的性质。我们将细节留待未来的工作。

Illustrative example: For the illustrative example (Figure 1 in the main paper) we rely on the following setting, taken from Shimodaira (2000); Sugiyama et al. (2007); You et al. (2019): The data points are labelled with

y = \frac{\sin (π x)}{π x}

with random noise sampled from the normal distribution

N (0, {(\frac{1}{4})}^{2})

. Moreover

p_{X} \sim N (1, \frac{1}{4}), q_{X} \sim N (2, {(\frac{1}{4})}^{2})

. The density ratio

β

can be computed analytically and is bounded. We aggregate several linear models with our approach and compare it to the optimal linear model, whose coefficients have been evaluated using a computer algebra system.

说明性示例：对于说明性示例（主论文中的图 1），我们依赖于以下设置，取自 Shimodaira (2000); Sugiyama 等人 (2007); You 等人 (2019)：数据点被标记为

y = \frac{\sin (π x)}{π x}

，并且随机噪声来自正态分布

N (0, {(\frac{1}{4})}^{2})

。此外

p_{X} \sim N (1, \frac{1}{4}), q_{X} \sim N (2, {(\frac{1}{4})}^{2})

。密度比

β

可以通过解析方式计算，并且是有界的。我们将多个线性模型与我们的方法聚合，并将其与最优线性模型进行比较，后者的系数是使用计算代数系统评估的。

Academic Dataset We rely on the Transformed Moons dataset (Zellinger et al., 2021), allowing us to visualize and address low-dimensional input data. The dataset consists of two-dimensional input data points forming two classes with a "moon-shaped" support. The shift from source to target domain is simulated by a transformation in input space as depicted in Figure 3. The results are shown in the following table:

学术数据集我们依赖于转化月亮数据集（Zellinger 等，2021），这使我们能够可视化和处理低维输入数据。该数据集由形成两个类别的二维输入数据点组成，具有“月亮形状”的支持。源域到目标域的转变通过输入空间中的变换进行模拟，如图 3 所示。结果显示在以下表格中：

Language Dataset To evaluate our method on a language task, we rely on the Amazon Reviews (Blitzer et al., 2006) dataset. This dataset consists of text reviews from four domains: books (B), DVDs (D), electronics (E), and kitchen appliances (K). Reviews are encoded in 5000 dimensional feature vectors of bag-of-words unigrams and bigrams with binary labels: label 0 if the product is obtain twelve domain adaptation tasks, where each category serves once as source domain and once as target domain (e.g., see Table 15). We follow similar data splits as previous works (Chen et al., 2012; Louizos et al., 2016; Ganin et al., 2016). In particular, we use 4000 labeled source examples and 4000 unlabeled target examples for training, and over 1000 examples for testing

语言数据集为了在语言任务上评估我们的方法，我们依赖于亚马逊评论（Blitzer 等，2006）数据集。该数据集由来自四个领域的文本评论组成：书籍（B）、DVD（D）、电子产品（E）和厨房电器（K）。评论被编码为5000维特征向量，包含单词袋的单元词和双元词，并带有二元标签：如果产品获得标签0，则进行十二个领域适应任务，其中每个类别作为源域和目标域各使用一次（例如，见表 15）。我们遵循与之前工作（Chen 等，2012；Louizos 等，2016；Ganin 等，2016）相似的数据划分。特别地，我们使用4000个标记的源示例和4000个未标记的目标示例进行训练，并使用超过1000个示例进行测试。

Method	Transformed Moons
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.994 (\pm 0.003)$	$0.994 (\pm 0.000)$	1.000(±0.003	$1.000 (\pm 0.003)$	$0.985 (\pm 0.012)$	$0.994 (\pm 0.003)$	$0.994 (\pm 0.003$	$0.992 (\pm 0.000)$	$0.998 (\pm 0.000)$
AdvSKM	$0.985 (\pm 0.000)$	$1.000 (\pm 0.000)$	0.998(±0.000)	$0.998 (\pm 0.000)$	$0.987 (\pm 0.014)$	$0.985 (\pm 0.000)$	$0.985 (\pm 0.000)$	$1.000 (\pm 0.000)$	1.000(±0.000)
DIRT	$0.964 (\pm 0.009)$	$0.827 (\pm 0.018)$	$0.830 (\pm 0.017)$	$0.981 (\pm 0.003)$	0.990(±0.006	0.964(±0.009)	$0.869 (\pm 0.009)$	0.983(±0.003)	0.966(±0.000)
DDC	$0.994 (\pm 0.023$	$0.998 (\pm 0.000$	$0.996 (\pm 0.000)$	$0.996 (\pm 0.000)$	$0.981 (\pm 0.009)$	$0.994 (\pm 0.023)$	$0.994 (\pm 0.023)$	$0.998 (\pm 0.000)$	$0.998 (\pm 0.000)$
CMD	$0.990 (\pm 0.006$	$0.956 (\pm 0.000)$	$0.964 (\pm 0.000)$	$0.996 (\pm 0.000$	$0.994 (\pm 0.006)$	$0.990 (\pm 0.006)$	$0.990 (\pm 0.006)$	$0.996 (\pm 0.000$	$1.000 (\pm 0.000)$
MMDA	0.990(±0.006)	1.000l±0.003	1.000(±0.003)	$1.000 (\pm 0.003)$	$0.996 (\pm {0.009}^{\circ}$	$0.990 (\pm 0.006)$	$0.990 \hat{i} \pm 0.006$	$1.000 (\pm 0.003)$	1.000(±0.003)
CoDATS	0.979(±0.016	1.000(±0.044	1.000(±0.052)	1.000(±0.016)	$0.981 (\pm 0.012)$	0.979(±0.016)	$0.979 (\pm 0.174)$	$1.000 (\pm {0.010}^{\circ})$	1.000(±0.045)
Deep-Coral	$1.000 (\pm 0.009)$	$1.000 (\pm {0.000}^{\circ})$	$1.000 (\pm 0.000)$	$1.000 (\pm 0.000)$	$0.985 (\pm 0.000)$	$1.000 (\pm 0.009)$	$1.000 (\pm {0.009}^{\circ})$	1.000(±0.000)	$1.000 (\pm 0.000)$
CDAN	1.000(±0.000)	$1.000 (\pm 0.000)$	1.000(±0.000)	1.000(±0.000)	$0.985 / \pm {0.026}^{\circ}$	1.000(±0.000)	1.000(±0.000’	1.000(±0.000	1.000(±0.000)
DANN	$0.994 (\pm 0.006)$	$1.000 \dot{0} \pm {0.000}^{\dot{3}}$	1.000/±0.000%	1.000/±0.000	$0.994 (\pm 0.016)$	$0.994 (\pm 0.006)$	$0.994 \hat{i} \hat{\pm 0.006}$	1.000(±0.003	1.000/±0.003
DSAN	$0.990 (\pm 0.009)$	$1.000 \dot{(} \pm 0.000 \dot{0}$	$1.000 (\pm 0.000)$	$1.000 (\pm 0.000)$	$1.000 (\pm 0.003)$	$0.990 (\pm 0.009)$	$0.990 (\pm 0.009)$	$1.000 \pm 0.000$	$1.000 (\pm 0.000)$
Avg.	$0.989 (\pm 0.008)$	$0.980 (\pm 0.006)$	$0.981 (\pm 0.007)$	$0.997 (\pm 0.002)$	$0.989 (\pm 0.010)$	$0.989 (\pm 0.008)$	$0.981 (\pm 0.022)$	$0.997 (\pm 0.002)$	$0.997 (\pm 0.005)$

方法	变换的月亮
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
HoMM	$0.994 (\pm 0.003)$	$0.994 (\pm 0.000)$	1.000(±0.003)	$1.000 (\pm 0.003)$	$0.985 (\pm 0.012)$	$0.994 (\pm 0.003)$	$0.994 (\pm 0.003$	$0.992 (\pm 0.000)$	$0.998 (\pm 0.000)$
AdvSKM	$0.985 (\pm 0.000)$	$1.000 (\pm 0.000)$	0.998(±0.000)	$0.998 (\pm 0.000)$	$0.987 (\pm 0.014)$	$0.985 (\pm 0.000)$	$0.985 (\pm 0.000)$	$1.000 (\pm 0.000)$	1.000(±0.000)
DIRT	$0.964 (\pm 0.009)$	$0.827 (\pm 0.018)$	$0.830 (\pm 0.017)$	$0.981 (\pm 0.003)$	0.990(±0.006)	0.964(±0.009)	$0.869 (\pm 0.009)$	0.983(±0.003)	0.966(±0.000)
DDC	$0.994 (\pm 0.023$	$0.998 (\pm 0.000$	$0.996 (\pm 0.000)$	$0.996 (\pm 0.000)$	$0.981 (\pm 0.009)$	$0.994 (\pm 0.023)$	$0.994 (\pm 0.023)$	$0.998 (\pm 0.000)$	$0.998 (\pm 0.000)$
CMD	$0.990 (\pm 0.006$	$0.956 (\pm 0.000)$	$0.964 (\pm 0.000)$	$0.996 (\pm 0.000$	$0.994 (\pm 0.006)$	$0.990 (\pm 0.006)$	$0.990 (\pm 0.006)$	$0.996 (\pm 0.000$	$1.000 (\pm 0.000)$
MMDA	0.990(±0.006)	1.000l±0.003	1.000(±0.003)	$1.000 (\pm 0.003)$	$0.996 (\pm {0.009}^{\circ}$	$0.990 (\pm 0.006)$	$0.990 \hat{i} \pm 0.006$	$1.000 (\pm 0.003)$	1.000(±0.003)
CoDATS	0.979(±0.016	1.000(±0.044	1.000(±0.052)	1.000(±0.016)	$0.981 (\pm 0.012)$	0.979(±0.016)	$0.979 (\pm 0.174)$	$1.000 (\pm {0.010}^{\circ})$	1.000(±0.045)
Deep-Coral	$1.000 (\pm 0.009)$	$1.000 (\pm {0.000}^{\circ})$	$1.000 (\pm 0.000)$	$1.000 (\pm 0.000)$	$0.985 (\pm 0.000)$	$1.000 (\pm 0.009)$	$1.000 (\pm {0.009}^{\circ})$	1.000(±0.000)	$1.000 (\pm 0.000)$
CDAN	1.000(±0.000)	$1.000 (\pm 0.000)$	1.000(±0.000)	1.000(±0.000)	$0.985 / \pm {0.026}^{\circ}$	1.000(±0.000)	1.000(±0.000’	1.000(±0.000	1.000(±0.000)
DANN	$0.994 (\pm 0.006)$	$1.000 \dot{0} \pm {0.000}^{\dot{3}}$	1.000/±0.000%	1.000/±0.000	$0.994 (\pm 0.016)$	$0.994 (\pm 0.006)$	$0.994 \hat{i} \hat{\pm 0.006}$	1.000(±0.003)	1.000/±0.003
DSAN	$0.990 (\pm 0.009)$	$1.000 \dot{(} \pm 0.000 \dot{0}$	$1.000 (\pm 0.000)$	$1.000 (\pm 0.000)$	$1.000 (\pm 0.003)$	$0.990 (\pm 0.009)$	$0.990 (\pm 0.009)$	$1.000 \pm 0.000$	$1.000 (\pm 0.000)$
平均值	$0.989 (\pm 0.008)$	$0.980 (\pm 0.006)$	$0.981 (\pm 0.007)$	$0.997 (\pm 0.002)$	$0.989 (\pm 0.010)$	$0.989 (\pm 0.008)$	$0.981 (\pm 0.022)$	$0.997 (\pm 0.002)$	$0.997 (\pm 0.005)$

Image Dataset Our third dataset is MiniDomainNet, which is based on the DomainNet-2019 dataset (Peng et al., 2019) consisting of six different image domains (Quickdraw: Q, Real: R, Clipart: C, Sketch: S, Infograph: I, and Painting: P). We follow Zellinger et al. (2021) and rely on the reduced version of DomainNet-2019, referred to as MiniDomainNet, which reduces the number of classes to the top-five largest representatives in the training set across all six domains. To further improve computation time, we rely on a ImageNet (Krizhevsky et al., 2012) pre-trained ResNet-18 (He et al., 2016) backbone. Therefore, we assume that the backbone has learned lower-level filters suitable for the "Real" image category, and we only need to adapt to the remaining five domains (e.g., Clipart, Sketch). This results in five domain adaptation tasks.

图像数据集我们的第三个数据集是 MiniDomainNet，它基于 DomainNet-2019 数据集（Peng et al., 2019），该数据集由六个不同的图像领域组成（Quickdraw: Q，Real: R，Clipart: C，Sketch: S，Infograph: I 和 Painting: P）。我们遵循 Zellinger et al. (2021) 的方法，依赖于 DomainNet-2019 的简化版本，称为 MiniDomainNet，该版本将类别数量减少到训练集中所有六个领域中前五个最大的代表。为了进一步提高计算时间，我们依赖于一个在 ImageNet（Krizhevsky et al., 2012）上预训练的 ResNet-18（He et al., 2016）主干网络。因此，我们假设主干网络已经学习了适合“真实”图像类别的低级滤波器，我们只需适应其余五个领域（例如，Clipart，Sketch）。这导致了五个领域适应任务。

Time-Series Dataset We based our time-series experiments on the four datasets included in the AdaTime benchmark suite (Ragab et al., 2022), which consists of UCI-HAR, WISDM, HHAR, and Sleep-EDF. The suite includes four representative datasets spanning 20 cross-domain real-world scenarios, i.e., human activity recognition and sleep stage classification. The first dataset is the Human Activity Recognition (HAR) (Anguita et al., 2013) dataset from the UC Irvine Repository denoted as UCI-HAR, which contains data from three motion sensors (accelerometer, gyroscope and body-worn sensors) gathered using smartphones from 30 different subjects. It classifies their activities in several categories, namely, walking, walking upstairs, downstairs, standing, sitting, and lying down. The WISDM (Kwapisz et al., 2011) dataset is a class-imbalanced variant from collected accelerometer sensors, including GPS data, from 29 different subjects which are performing similar activities as in the UCI-HAR dataset. The Heterogeneity Human Activity Recognition (HHAR) (Stisen et al., 2015) dataset investigate sensor-, device- and workload-specific heterogeneities using 36 smartphones and smartwatches, consisting of 13 different device models from four manufacturers. Finally, the Sleep Stage Classification time-series setting aims to classify the electroencephalography

时间序列数据集我们的时间序列实验基于 AdaTime 基准套件中包含的四个数据集（Ragab et al., 2022），该套件包括 UCI-HAR、WISDM、HHAR 和 Sleep-EDF。该套件包含四个代表性数据集，涵盖 20 个跨领域的真实场景，即人类活动识别和睡眠阶段分类。第一个数据集是来自加州大学欧文分校库的人类活动识别（HAR）数据集（Anguita et al., 2013），标记为 UCI-HAR，包含来自 30 个不同受试者的三种运动传感器（加速度计、陀螺仪和身体佩戴传感器）收集的数据。它将他们的活动分类为几类，即走路、上楼、下楼、站立、坐着和躺下。WISDM（Kwapisz et al., 2011）数据集是一个类别不平衡的变体，来自 29 个不同受试者收集的加速度计传感器，包括 GPS 数据，这些受试者执行的活动与 UCI-HAR 数据集中的活动相似。异质人类活动识别（HHAR）（Stisen et al., 2015）数据集使用 36 部智能手机和智能手表，涵盖来自四个制造商的 13 种不同设备型号，研究传感器、设备和工作负载特定的异质性。最后，睡眠阶段分类时间序列设置旨在对脑电图进行分类。

(EEG) signals into five stages i.e., Wake (W), Non-Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM). Analogous to Ragab et al. (2022); Eldele et al. (2021), we adopt the Sleep-EDF-20 dataset obtained from PhysioBank (Goldberger et al., 2000), which contains EEG readings from 20 healthy subjects. For all datasets, each subject is treated as an own domain, and adopt from a source subject to a target subject.

(EEG) 信号分为五个阶段，即清醒 (W)、非快速眼动阶段 (N1, N2, N3) 和快速眼动 (REM)。类似于 Ragab 等人 (2022)；Eldele 等人 (2021)，我们采用了从 PhysioBank (Goldberger 等人, 2000) 获取的 Sleep-EDF-20 数据集，该数据集包含来自 20 名健康受试者的 EEG 读数。对于所有数据集，每个受试者被视为一个独立的领域，并从源受试者适应到目标受试者。

This section is meant to provide further details on the overall computational setting of our experiments. We start by giving an overview on the used computational resources for the specific datasets and the implementation tools. Next, we describe the network architectures for the individual datasets in greater detail. In the third subsection we elaborate on the construction of our models, and the fourth subsection is devoted to matrix inversion. Finally, in the last subsection, we describe the detailed empirical results and give the complete tables.

本节旨在提供有关我们实验整体计算设置的进一步细节。我们首先概述用于特定数据集的计算资源和实现工具。接下来，我们更详细地描述各个数据集的网络架构。在第三小节中，我们阐述了模型的构建，第四小节专门讨论矩阵求逆。最后，在最后一小节中，我们描述详细的实证结果并给出完整的表格。

Overall, to compute the results in our tables, we trained 16680 models with an approximate computational budget of 1500 GPU/hours on one high-performance computing station using

8 \times

NVIDIA P100 16GB, 512GB RAM, 40 Cores Xeon(R) CPU E5-2698 v4 @ 2.20GHz on CentOS Linux 7.

总体而言，为了计算我们表格中的结果，我们在一台高性能计算站上训练了 16680 个模型，计算预算约为 1500 GPU/小时，使用

8 \times

NVIDIA P100 16GB、512GB RAM、40 核 Xeon(R) CPU E5-2698 v4 @ 2.20GHz，操作系统为 CentOS Linux 7。

All methods have been implemented in Python using the Pytorch (Paszke et al., 2017, BSD license) library. For monitoring the runs we used Weights & Biases (Biewald, 2020, MIT license). We use Scikit-learn (Pedregosa et al., 2011) library for evaluation measures and toy datasets, and the TQDM (da Costa-Luis, 2019) library, and Tensorboard (Abadi et al., 2015) for keeping track of the progress of our experiments. We built parts of our implementation on the codebase of Zellinger et al. (2021, MIT License) and Ragab et al. (2022, MIT License).

所有方法均已使用 Pytorch (Paszke et al., 2017, BSD 许可证) 库在 Python 中实现。为了监控运行情况，我们使用了 Weights & Biases (Biewald, 2020, MIT 许可证)。我们使用 Scikit-learn (Pedregosa et al., 2011) 库进行评估指标和玩具数据集，并使用 TQDM (da Costa-Luis, 2019) 库和 Tensorboard (Abadi et al., 2015) 来跟踪实验进展。我们的实现部分基于 Zellinger et al. (2021, MIT 许可证) 和 Ragab et al. (2022, MIT 许可证) 的代码库构建。

In this subsection, we provide details on the model architectures and the training setup for every dataset. Our base architectures are based on the AdaTime benchmark suite, which is a large-scale evaluation of domain adaptation algorithms on time-series data. We extended the benchmark suite to support 11 state-of-the-art model architectures on multiple dataset types ranging from language, image to time-series data, addressed by Transformed Moons, Amazon Reviews, MiniDomainNet and the four time-series datasets (UCI-HAR, WISDM, HHAR, and Sleep-EDF) spanning in total 38 cross-domain real-world scenarios.

在本小节中，我们提供每个数据集的模型架构和训练设置的详细信息。我们的基础架构基于 AdaTime 基准套件，这是对时间序列数据上的领域适应算法的大规模评估。我们扩展了基准套件，以支持 11 种最先进的模型架构，涵盖从语言、图像到时间序列数据的多种数据集类型，涉及 Transformed Moons、Amazon Reviews、MiniDomainNet 和四个时间序列数据集（UCI-HAR、WISDM、HHAR 和 Sleep-EDF），总共涵盖 38 个跨领域的真实场景。

512 dimensional extracted features. To alleviate overfitting effects on pre-computed features, we perform data augmentation on the images of each batch and forward each batch through the backbone. We incorporate zero padding before resizing the images to

256 \times 256

to avoid image distortions. Furthermore, in alignment with data augmentation techniques from Shorten & Khoshgoftaar (2019), we perform random resized cropping to

224 \times 224

with a random viewport between

70 %

and

100 %

of the original image,random horizontal flipping,color jittering of

0.25 %

on each RGB channel,and a

\pm 2

degree rotation.

512 维特征上进行后续计算。为了减轻对预计算特征的过拟合影响，我们对每个批次的图像进行数据增强，并将每个批次通过主干网络进行前向传播。我们在调整图像大小到

256 \times 256

之前加入零填充，以避免图像失真。此外，结合 Shorten & Khoshgoftaar（2019）的数据增强技术，我们对原始图像进行随机调整裁剪到

224 \times 224

，随机视口在

70 %

和

100 %

之间，随机水平翻转，对每个 RGB 通道进行

0.25 %

的颜色抖动，以及

\pm 2

度的旋转。

After the ResNet-18 backbone output, we add a projection layer, and define the domain adaptation layers on which we use the domain adaptation methods to align the representations. The backbone and projection layers are defined as a common architecture across the different domain adaptation methods. Additional layers are further added for the classification networks, according to the requirements of the individual domain adaptation methods (e.g., CMD, HoMM). The number of layers/neurons in the upper layers of our architecture have been tuned in order to achieve the best performance in the source-only setup. See Table 5 for a detailed description of the architecture used. We perform listed methods, and with 3 repetitions based on different random weights initialization. All class prediction models have been trained for 60 epochs and domain classifiers for 100 epochs with Adam optimizer,a learning rate of

0.001, β_{1} = 0.9, β_{2} = 0.999

,batchsize of 128 and weight decay of 0.0001 Additional hyper-parameters are reported in Table 9.

在 ResNet-18 主干输出后，我们添加了一个投影层，并定义了领域适应层，在这些层上我们使用领域适应方法来对齐表示。主干和投影层被定义为不同领域适应方法之间的通用架构。根据各个领域适应方法的要求（例如，CMD，HoMM），进一步添加了额外的层用于分类网络。我们架构上层的层数/神经元数量经过调优，以便在仅源数据的设置中实现最佳性能。有关所用架构的详细描述，请参见表 5。我们执行了列出的各种方法，并基于不同的随机权重初始化进行了 3 次重复。所有类别预测模型已训练 60 个周期，领域分类器训练 100 个周期，使用 Adam 优化器，学习率为

0.001, β_{1} = 0.9, β_{2} = 0.999

，批量大小为 128，权重衰减为 0.0001。其他超参数在表 9 中报告。

AdaTime Unless stated otherwise, we follow the implementation and hyper-parameter settings as reported in Ragab et al. (2022). We extended the AdaTime suite to comprise a collection of 11 domain adaptation algorithms. We learned all domain adaptations models according to the following approaches (see also Table 6, Table 7 and Table 8): Deep Domain Confusion (DDC) (Tzeng et al., 2014), Correlation Alignment via Deep Neural Networks (Deep-Coral) (Sun et al., 2017), Higher-order Moment Matching (HoMM) (Chen et al., 2020), Minimum Discrepancy Estimation for Deep Domain Adaptation (MMDA) (Rahman et al., 2020), Central Moment Discrepancy (CMD) (Zellinger et al., 2017), Deep Subdomain Adaptation (DSAN) (Zhu et al., 2021), Domain-Adversarial Neural Networks (DANN) (Ganin et al., 2016), Conditional Adversarial Domain Adaptation (CDAN) (Long et al., 2018), A DIRT-T Approach to Unsupervised Domain Adaptation (DIRT) (Shu et al., 2018), Convolutional deep Domain Adaptation model for Time-Series data (CoDATS) (Wilson et al., 2020), and Adversarial Spectral Kernel Matching (AdvSKM) (Liu & Xue, 2021). The backbone architecture of all models is a 1D-CNN network. It consists of three CNN blocks and each block has a 1D pooling and Dropout. In the first block, the kernel size of the convolutional layer is set according to the dataset as reported in Ragab et al. (2022). After the convolutional blocks, we apply an 1D adaptive pooling layer. All methods are trained for 100 epochs on all datasets. The batch size is 32, except for Sleep-EDF, where we use batch size of 128. All models are trained with Adam optimizer (Kingma & Ba,2014) and weight decay of

10^{- 4}

. Additional hyper-parameters are reported in Table 10.

AdaTime 除非另有说明，否则我们遵循 Ragab 等人（2022）中报告的实现和超参数设置。我们扩展了 AdaTime 套件，以包含 11 种领域适应算法的集合。我们根据以下方法学习了所有领域适应模型（另见表 6、表 7 和表 8）：深度领域混淆（DDC）（Tzeng 等人，2014），通过深度神经网络的相关性对齐（Deep-Coral）（Sun 等人，2017），高阶矩匹配（HoMM）（Chen 等人，2020），深度领域适应的最小差异估计（MMDA）（Rahman 等人，2020），中心矩差异（CMD）（Zellinger 等人，2017），深度子领域适应（DSAN）（Zhu 等人，2021），领域对抗神经网络（DANN）（Ganin 等人，2016），条件对抗领域适应（CDAN）（Long 等人，2018），无监督领域适应的 DIRT-T 方法（DIRT）（Shu 等人，2018），时间序列数据的卷积深度领域适应模型（CoDATS）（Wilson 等人，2020），以及对抗谱核匹配（AdvSKM）（Liu & Xue，2021）。所有模型的主干架构是一个 1D-CNN 网络。它由三个 CNN 块组成，每个块都有一个 1D 池化和 Dropout。在第一个块中，卷积层的核大小根据 Ragab 等人（2022）中报告的数据集进行设置。在卷积块之后，我们应用一个 1D 自适应池化层。所有方法在所有数据集上训练 100 个周期。批量大小为 32，除了 Sleep-EDF，我们使用批量大小为 128。所有模型均使用 Adam 优化器（Kingma & Ba，2014）和权重衰减

10^{- 4}

进行训练。其他超参数在表 10 中报告。

Our algorithm, IWA, constructs an ensemble from a sequence of different classifiers, e.g. obtained from a sequence of possible hyper-parameter configurations in domain adaptation algorithms. To all datasets with different hyper-parameter choices. For the experiments on the language, im-

我们的算法 IWA 从一系列不同的分类器构建一个集成，例如从领域适应算法中可能的超参数配置序列中获得。对于所有具有不同超参数选择的数据集。关于语言的实验，im-

Matrix inversion is a well-known numerical task, especially in cases of limited computing precision and ill-conditioned matrices. In our case, similar models in the given sequence can cause numerical instability due to limited compute precision. That is, occasionally a tabula rasa inversion of the matrix

\tilde{G}

in Algorithm 1 is numerically unstable. Various standard approaches can be applied to handle this common issue, including the exclusion of similar models and various regularization techniques. In our computational setup, we rely on the Python routine numpy.linalg.pinv, which is based on the eigendecompostion of

\tilde{G}

(coinciding with the singular value decomposition in our case due to positive-definiteness) and an eigenvalue-based regularization based on a treshold value rcond for small eigenvalues, see Strang (1980, pages 138-140) for details. The choice of rcond depends on the scale of the Gram matrix and can therefore be chosen by source data only. Based on evaluating our method on source data only (target domain is fixed to be source domain) on several choices for rcond, we obtain a stable choice for rcond of

10^{- 1}

for all datasets.

矩阵求逆是一个众所周知的数值任务，尤其是在计算精度有限和病态矩阵的情况下。在我们的案例中，给定序列中的相似模型可能会由于计算精度有限而导致数值不稳定。也就是说，在算法 1 中，矩阵

\tilde{G}

的偶尔“白板”求逆在数值上是不稳定的。可以应用多种标准方法来处理这一常见问题，包括排除相似模型和各种正则化技术。在我们的计算设置中，我们依赖于基于

\tilde{G}

的特征分解的 Python 程序 numpy.linalg.pinv（在我们的案例中由于正定性与奇异值分解相一致）以及基于小特征值的阈值 rcond 的特征值正则化，详细信息请参见 Strang (1980, 第 138-140 页)。rcond 的选择取决于 Gram 矩阵的规模，因此可以仅通过源数据进行选择。基于对源数据（目标领域固定为源领域）上 rcond 的多个选择评估我们的方法，我们为所有数据集获得了 rcond 的稳定选择

10^{- 1}

。

Figure 2 in the main paper suggests that there is a positive correlation between the target accuracy of each individual model (orange dashed line, Figure 2 top) and the respective aggregation weight (red bars, Figure 2 bottom), if the linear aggregation of the models is computed by our method IWA. In the following, we analyze whether this trend holds throughout all other experiments.

主论文中的图 2 表明，如果通过我们的方法 IWA 计算模型的线性聚合，则每个单独模型的目标准确度（橙色虚线，图 2 顶部）与相应的聚合权重（红色条，图 2 底部）之间存在正相关。接下来，我们分析这一趋势是否在所有其他实验中都成立。

More precisely,for a given sequence

f_{1}, \dots, f_{l}

of models,we compute the Pearson correlation coefficient between the aggregation weights

{\tilde{c}}_{1}, \dots, {\tilde{c}}_{l}

(see Algorithm 1) and the corresponding target accuracies

\frac{1}{t} \sum_{i = 1}^{t} 1 [y_{i} = f_{1} (x_{i})], \dots, \frac{1}{t} \sum_{i = 1}^{t} 1 [y_{i} = f_{l} (x_{i})]

for the target test data

(x_{1}, y_{1}), \dots, (x_{t}, y_{t})

,where

1 [P] = 1

iff

P

is true and

1 [P] = 0

otherwise. In this context,a positive correlation coefficient means that the aggregation algorithm assigns a higher weight to models performing better on target samples. We calculate these coefficients for our method IWA and the other linear regression baselines SOR, TCR, and TMR for all domain adaptation methods across all datasets and cross-domain scenarios. Note that for this analysis we cannot compare to the other baseline TMV, as the count-based aggregation by majority voting does not involve the computation of aggregation weights.

更准确地说，对于给定的模型序列

f_{1}, \dots, f_{l}

，我们计算聚合权重

{\tilde{c}}_{1}, \dots, {\tilde{c}}_{l}

（见算法 1）与目标测试数据

(x_{1}, y_{1}), \dots, (x_{t}, y_{t})

的相应目标准确率

\frac{1}{t} \sum_{i = 1}^{t} 1 [y_{i} = f_{1} (x_{i})], \dots, \frac{1}{t} \sum_{i = 1}^{t} 1 [y_{i} = f_{l} (x_{i})]

之间的 Pearson 相关系数，其中

1 [P] = 1

当且仅当

P

为真时，

1 [P] = 0

否则。在此背景下，正相关系数意味着聚合算法为在目标样本上表现更好的模型分配了更高的权重。我们为我们的方法 IWA 和其他线性回归基线 SOR、TCR 和 TMR 计算这些系数，涵盖所有领域适应方法、所有数据集和跨域场景。请注意，在此分析中，我们无法与其他基线 TMV 进行比较，因为基于多数投票的计数聚合不涉及聚合权重的计算。

We study the sensitivity of our method with respect to adding inaccurate models in the given sequence of models. We define an innacurate model as having a target accuracy lower than

80 %

of the target accuracy of the model computed without domain adaptation (SO). In particular, we add +10, +50 and +100 inaccurate models to the given sequences of models. One inaccurate model is constructed as follows: First, a model is chosen uniformly at random from the given sequence of models. Second, the outputs of the chosen model are corrupted by adding, to half of the elements of its vector-valued output, random Gaussian noise with zero-mean and unit variance.

我们研究了我们的方法在添加不准确模型时的敏感性，考虑给定的模型序列。我们将不准确模型定义为其目标准确率低于

80 %

的模型，该模型是在没有领域适应（SO）的情况下计算的目标准确率。具体而言，我们向给定的模型序列中添加 +10、+50 和 +100 个不准确模型。一个不准确模型的构造过程如下：首先，从给定的模型序列中随机均匀选择一个模型。其次，通过向其向量值输出的一半元素添加均值为零、方差为一的随机高斯噪声来破坏所选模型的输出。

In Figure 7, we show the median performance over all domain adaptation methods for each dataset. We see, that the performance of our method (IWA) is not very sensitive w.r.t. an increase in the number of inaccurate models. This is in contrast to the heuristics (e.g. TMV, TMR or TCR) and model selection method DEV, which show a high sensitivity concerning adding inaccurate models. SOR also shows stable results; it is, however, clearly outperformed by our method in five out of six stable choice concerning inaccurate models in the given sequence.

在图7中，我们展示了所有领域适应方法在每个数据集上的中位性能。我们看到，我们的方法（IWA）对不准确模型数量增加的敏感性并不高。这与启发式方法（例如TMV、TMR或TCR）和模型选择方法DEV形成对比，后者对添加不准确模型表现出高度敏感性。SOR也显示出稳定的结果；然而，在给定序列中，对于不准确模型的六个稳定选择中的五个，明显被我们的方法超越。

Table 11: Average target accuracies (and average standard deviations) for all 7 datasets (e.g., Sleep-EDF, MiniDomainNet, Amazon Reviews) taken over several domain adaptation tasks (e.g., 5 on Sleep-EDF, 5 on MiniDomainNet, 12 on Amazon Reviews), 11 domain adaptation methods (e.g., DANN, HoMM, CMD) and 3 repetitions with different random initialization of model weights. The input sequences of the approaches (e.g., DEV, IWA) consist of neural networks computed by runs of the domain adaptation methods with different hyper-parameters (e.g.,8 different values of

λ

for

表 11：在多个领域适应任务（例如，在 Sleep-EDF 上进行 5 次，在 MiniDomainNet 上进行 5 次，在 Amazon Reviews 上进行 12 次）、11 种领域适应方法（例如，DANN、HoMM、CMD）以及 3 次不同随机初始化模型权重的重复实验中，所有 7 个数据集（例如，Sleep-EDF、MiniDomainNet、Amazon Reviews）的平均目标准确率（及其平均标准差）。

Dataset		Heuristic				Theoretical error guarantees
Dataset	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
Transformed Moons	$0.989 (\pm 0.008)$	$0.980 (\pm 0.006)$	$0.981 (\pm 0.007)$	$0.997 (\pm 0.002)$	$0.989 (\pm 0.010)$	$0.989 (\pm 0.008)$	$0.981 (\pm 0.022)$	$0.997 (\pm 0.002)$	$0.997 (\pm 0.005)$
Amazon Reviews	$0.767 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.010)$	0.789(±0.010	$0.772 (\pm 0.014)$	$0.764 (\pm 0.019)$	$0.788 (\pm 0.009$	$0.781 (\pm 0.012)$
MiniDomainNet	$0.507 (\pm 0.022)$	0.526(±0.011	$0.525 (\pm 0.014)$	$0.526 (\pm 0.013)$	$0.518 (\pm 0.012)$	$0.513 (\pm 0.022)$	$0.515 (\pm 0.028)$	0.531(±0.011	$0.534 (\pm 0.022)$
Sleep-EDF	$0.655 (\pm 0.054)$	$0.729 (\pm 0.018)$	$0.729 (\pm 0.024)$	$0.725 (\pm 0.023)$	$0.717 (\pm 0.028)$	$0.700 (\pm 0.052)$	$0.660 (\pm 0.057)$	$0.737 (\pm 0.020)$	$0.712 (\pm 0.045)$
UCI-HAR	$0.770 (\pm 0.046)$	$0.840 (\pm 0.017)$	$0.833 (\pm 0.023)$	$0.832 (\pm 0.024)$	$0.769 (\pm 0.060)$	$0.774 (\pm {0.070}^{\circ})$	$0.765 (\pm 0.090)$	$0.835 (\pm 0.020)$	$0.850 \dot{} \dot{} \dot{} \dot{} \dot{} 0.029 \dot{}$
HHAR	$0.732 (\pm 0.042)$	$0.771 (\pm 0.015)$	$0.768 (\pm 0.017)$	0.7711+0.018	0.722(+0.068)	$0.746 (\pm {0.037}^{\circ})$	$0.722 (\pm 0.063)$	$0.787 (\pm 0.012)$	$0.784 (\pm 0.028)$
WISDM	$0.736 \hat{i} + 0.050 \hat{j}$	0.768 (±0.027	$0.768 (\pm 0.036)$	$0.765 (\pm 0.037)$	$0.737 (\pm 0.062)$	$0.736 (\pm 0.052)$	$0.726 (\pm 0.077)$	$0.764 (\pm 0.025)$	$0.771 (\pm 0.046)$

数据集		启发式				理论误差保证
数据集	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA (我们的)	TB
转变的月球	$0.989 (\pm 0.008)$	$0.980 (\pm 0.006)$	$0.981 (\pm 0.007)$	$0.997 (\pm 0.002)$	$0.989 (\pm 0.010)$	$0.989 (\pm 0.008)$	$0.981 (\pm 0.022)$	$0.997 (\pm 0.002)$	$0.997 (\pm 0.005)$
亚马逊评论	$0.767 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.010)$	0.789(±0.010	$0.772 (\pm 0.014)$	$0.764 (\pm 0.019)$	$0.788 (\pm 0.009$	$0.781 (\pm 0.012)$
MiniDomainNet	$0.507 (\pm 0.022)$	0.526(±0.011	$0.525 (\pm 0.014)$	$0.526 (\pm 0.013)$	$0.518 (\pm 0.012)$	$0.513 (\pm 0.022)$	$0.515 (\pm 0.028)$	0.531(±0.011	$0.534 (\pm 0.022)$
Sleep-EDF	$0.655 (\pm 0.054)$	$0.729 (\pm 0.018)$	$0.729 (\pm 0.024)$	$0.725 (\pm 0.023)$	$0.717 (\pm 0.028)$	$0.700 (\pm 0.052)$	$0.660 (\pm 0.057)$	$0.737 (\pm 0.020)$	$0.712 (\pm 0.045)$
UCI-HAR	$0.770 (\pm 0.046)$	$0.840 (\pm 0.017)$	$0.833 (\pm 0.023)$	$0.832 (\pm 0.024)$	$0.769 (\pm 0.060)$	$0.774 (\pm {0.070}^{\circ})$	$0.765 (\pm 0.090)$	$0.835 (\pm 0.020)$	$0.850 \dot{} \dot{} \dot{} \dot{} \dot{} 0.029 \dot{}$
HHAR	$0.732 (\pm 0.042)$	$0.771 (\pm 0.015)$	$0.768 (\pm 0.017)$	0.7711+0.018	0.722(+0.068)	$0.746 (\pm {0.037}^{\circ})$	$0.722 (\pm 0.063)$	$0.787 (\pm 0.012)$	$0.784 (\pm 0.028)$
WISDM	$0.736 \hat{i} + 0.050 \hat{j}$	0.768 (±0.027	$0.768 (\pm 0.036)$	$0.765 (\pm 0.037)$	$0.737 (\pm 0.062)$	$0.736 (\pm 0.052)$	$0.726 (\pm 0.077)$	$0.764 (\pm 0.025)$	$0.771 (\pm 0.046)$

Method	Amazon Reviews
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.769 (\pm 0.009)$	$0.777 (\pm 0.010)$	$0.778 (\pm 0.010)$	$0.778 (\pm 0.011)$	$0.777 (\pm 0.010)$	$0.765 (\pm 0.011)$	$0.766 (\pm 0.011)$	$0.778 (\pm 0.010)$	$0.769 (\pm 0.012)$
AdvSKM	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009)$	$0.779 (\pm 0.010)$	$0.779 (\pm 0.008)$	$0.778 (\pm 0.011)$	$0.769 (\pm 0.012)$	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009)$	$0.770 (\pm 0.012)$
DIRT	$0.764 (\pm 0.009)$	$0.786 (\pm 0.008)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.008)$	$0.800 (\pm 0.008$	$0.778 (\pm 0.022)$	$0.773 (\pm 0.056)$	$0.787 (\pm 0.008)$	$0.786 (\pm 0.009)$
DDC	$0.766 (\pm 0.012)$	0.779(±0.010	$0.780 (\pm 0.009$	$0.779 (\pm 0.010)$	$0.778 (\pm 0.010)$	$0.767 (\pm {0.017}^{\circ})$	$0.768 (\pm 0.011)$	$0.780 (\pm 0.010)$	$0.770 (\pm 0.013)$
CMD	$0.767 (\pm 0.012)$	$0.791 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.789 (\pm 0.010)$	$0.792 (\pm 0.010$	$0.765 (\pm 0.015)$	$0.710 (\pm 0.015)$	$0.794 (\pm 0.009$	$0.785 (\pm 0.009)$
MMDA	$0.767 (\pm 0.011)$	$0.787 (\pm 0.011)$	$0.785 (\pm 0.010)$	$0.785 (\pm 0.010)$	$0.787 (\pm 0.012)$	$0.769 (\pm 0.011)$	$0.766 (\pm 0.010)$	$0.787 (\pm 0.011)$	$0.782 (\pm 0.011)$
CoDATS	$0.766 (\pm 0.013)$	$0.795 (\pm 0.009)$	$0.793 (\pm 0.010)$	$0.794 (\pm 0.012)$	$0.799 (\pm 0.010)$	$0.779 (\pm 0.016)$	$0.773 (\pm 0.020)$	$0.796 (\pm 0.009$	$0.791 (\pm 0.015)$
Deep-Coral	$0.766 (\pm 0.012)$	$0.784 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.782 (\pm 0.009)$	$0.769 (\pm 0.016)$	$0.769 (\pm 0.037)$	$0.785 (\pm 0.009)$	$0.776 (\pm 0.013)$
CDAN	$0.767 (\pm 0.012)$	$0.788 (\pm 0.010$	$0.787 (\pm 0.009)$	$0.787 (\pm 0.010)$	$0.787 (\pm 0.011)$	$0.775 (\pm 0.011)$	$0.776 (\pm 0.014)$	$0.788 (\pm 0.010)$	$0.777 (\pm 0.011)$
DANN	$0.767 (\pm 0.012)$	$0.796 (\pm 0.010)$	$0.792 (\pm 0.010)$	$0.793 (\pm 0.010)$	$0.800 (\pm 0.011)$	$0.776 (\pm 0.011)$	$0.778 (\pm 0.012)$	$0.797 (\pm 0.009)$	$0.798 (\pm 0.012)$
DSAN	$0.769 (\pm 0.009)$	$0.796 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.791 (\pm 0.010)$	$0.800 (\pm 0.010)$	$0.779 (\pm 0.012)$	$0.763 (\pm 0.017)$	$0.795 (\pm 0.009)$	$0.789 (\pm 0.012)$
Avg.	$0.767 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.010)$	$0.789 (\pm 0.010)$	$0.772 (\pm 0.014)$	$0.764 (\pm 0.019)$	$0.788 (\pm 0.009)$	$0.781 (\pm 0.012)$

方法	亚马逊评论
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.769 (\pm 0.009)$	$0.777 (\pm 0.010)$	$0.778 (\pm 0.010)$	$0.778 (\pm 0.011)$	$0.777 (\pm 0.010)$	$0.765 (\pm 0.011)$	$0.766 (\pm 0.011)$	$0.778 (\pm 0.010)$	$0.769 (\pm 0.012)$
AdvSKM	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009)$	$0.779 (\pm 0.010)$	$0.779 (\pm 0.008)$	$0.778 (\pm 0.011)$	$0.769 (\pm 0.012)$	$0.766 (\pm 0.012)$	$0.780 (\pm 0.009)$	$0.770 (\pm 0.012)$
DIRT	$0.764 (\pm 0.009)$	$0.786 (\pm 0.008)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.008)$	$0.800 (\pm 0.008$	$0.778 (\pm 0.022)$	$0.773 (\pm 0.056)$	$0.787 (\pm 0.008)$	$0.786 (\pm 0.009)$
DDC	$0.766 (\pm 0.012)$	0.779（±0.010）	$0.780 (\pm 0.009$	$0.779 (\pm 0.010)$	$0.778 (\pm 0.010)$	$0.767 (\pm {0.017}^{\circ})$	$0.768 (\pm 0.011)$	$0.780 (\pm 0.010)$	$0.770 (\pm 0.013)$
CMD	$0.767 (\pm 0.012)$	$0.791 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.789 (\pm 0.010)$	$0.792 (\pm 0.010$	$0.765 (\pm 0.015)$	$0.710 (\pm 0.015)$	$0.794 (\pm 0.009$	$0.785 (\pm 0.009)$
MMDA	$0.767 (\pm 0.011)$	$0.787 (\pm 0.011)$	$0.785 (\pm 0.010)$	$0.785 (\pm 0.010)$	$0.787 (\pm 0.012)$	$0.769 (\pm 0.011)$	$0.766 (\pm 0.010)$	$0.787 (\pm 0.011)$	$0.782 (\pm 0.011)$
CoDATS	$0.766 (\pm 0.013)$	$0.795 (\pm 0.009)$	$0.793 (\pm 0.010)$	$0.794 (\pm 0.012)$	$0.799 (\pm 0.010)$	$0.779 (\pm 0.016)$	$0.773 (\pm 0.020)$	$0.796 (\pm 0.009$	$0.791 (\pm 0.015)$
Deep-Coral	$0.766 (\pm 0.012)$	$0.784 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.782 (\pm 0.009)$	$0.769 (\pm 0.016)$	$0.769 (\pm 0.037)$	$0.785 (\pm 0.009)$	$0.776 (\pm 0.013)$
CDAN	$0.767 (\pm 0.012)$	$0.788 (\pm 0.010$	$0.787 (\pm 0.009)$	$0.787 (\pm 0.010)$	$0.787 (\pm 0.011)$	$0.775 (\pm 0.011)$	$0.776 (\pm 0.014)$	$0.788 (\pm 0.010)$	$0.777 (\pm 0.011)$
DANN	$0.767 (\pm 0.012)$	$0.796 (\pm 0.010)$	$0.792 (\pm 0.010)$	$0.793 (\pm 0.010)$	$0.800 (\pm 0.011)$	$0.776 (\pm 0.011)$	$0.778 (\pm 0.012)$	$0.797 (\pm 0.009)$	$0.798 (\pm 0.012)$
DSAN	$0.769 (\pm 0.009)$	$0.796 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.791 (\pm 0.010)$	$0.800 (\pm 0.010)$	$0.779 (\pm 0.012)$	$0.763 (\pm 0.017)$	$0.795 (\pm 0.009)$	$0.789 (\pm 0.012)$
平均	$0.767 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.786 (\pm 0.010)$	$0.786 (\pm 0.010)$	$0.789 (\pm 0.010)$	$0.772 (\pm 0.014)$	$0.764 (\pm 0.019)$	$0.788 (\pm 0.009)$	$0.781 (\pm 0.012)$

Method	MiniDomainNet
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.509 (\pm 0.018)$	$0.526 (\pm 0.009)$	$0.524 (\pm 0.010)$	$0.523 (\pm 0.010)$	$0.518 (\pm 0.015)$	$0.511 (\pm 0.018)$	$0.511 (\pm 0.018)$	$0.531 (\pm 0.007)$	$0.537 (\pm 0.020)$
AdvSKM	$0.509 (\pm 0.015)$	$0.516 (\pm 0.011$	$0.514 (\pm 0.014)$	$0.514 (\pm 0.017)$	$0.514 (\pm 0.009)$	$0.515 (\pm 0.032)$	$0.512 (\pm 0.032)$	$0.516 (\pm 0.011)$	$0.522 (\pm 0.017)$
DIRT	$0.499 (\pm 0.022)$	$0.517 (\pm 0.006)$	$0.515 (\pm 0.009)$	$0.508 (\pm 0.006)$	$0.507 (\pm 0.018$	0.493(±0.028)	$0.498 (\pm 0.033)$	$0.519 (\pm 0.006$	$0.525 (\pm 0.020)$
DDC	$0.510 (\pm 0.032)$	0.514(±0.012	$0.512 (\pm 0.018)$	$0.511 (\pm 0.019$	$0.514 (\pm 0.015)$	$0.511 (\pm 0.028)$	$0.512 (\pm 0.043$	$0.516 (\pm 0.015$	$0.521 (\pm 0.019)$
CMD	$0.509 (\pm 0.019)$	$0.528 (\pm 0.011)$	$0.531 (\pm 0.016$	$0.531 (\pm {0.016}^{\circ}$	$0.522 (\pm 0.013$	$0.518 (\pm 0.025)$	$0.490 (\pm 0.037)$	$0.533 (\pm 0.009$	$0.533 (\pm 0.024)$
MMDA	$0.509 (\pm 0.021)$	0.524(±0.004	$0.522 (\pm 0.005)$	$0.523 (\pm 0.007)$	$0.519 (\pm 0.012)$	$0.517 (\pm 0.008)$	$0.524 (\pm 0.006)$	$0.527 (\pm 0.009$	$0.531 (\pm 0.029)$
CoDATS	0.502(±0.026)	$0.535 (\pm 0.026)$	$0.533 (\pm 0.027)$	$0.536 (\pm 0.028$	$0.525 (\pm 0.015)$	$0.518 (\pm 0.031)$	0.524(±0.051	$0.536 (\pm 0.025)$	$0.529 (\pm 0.042)$
Deep-Coral	$0.505 (\pm 0.022)$	$0.538 (\pm 0.009)$	$0.534 (\pm 0.013)$	$0.533 (\pm 0.012)$	$0.520 (\pm 0.010$	$0.518 (\pm 0.022)$	$0.524 (\pm 0.034$	$0.539 (\pm 0.012$	$0.535 (\pm 0.024)$
CDAN	$0.514 (\pm 0.028)$	$0.526 (\pm 0.013$	$0.524 (\pm 0.011)$	$0.525 (\pm 0.008)$	$0.517 (\pm 0.005)$	$0.513 (\pm 0.011)$	$0.516 (\pm 0.014)$	$0.532 (\pm 0.010$	$0.542 (\pm 0.017)$
DANN	$0.496 (\pm 0.023)$	$0.530 (\pm 0.011)$	$0.535 (\pm 0.011)$	$0.538 (\pm 0.005)$	$0.516 (\pm 0.011)$	$0.519 (\pm 0.015)$	$0.515 (\pm 0.013)$	$0.541 (\pm 0.006)$	$0.532 (\pm 0.014)$
DSAN	$0.509 (\pm 0.022)$	$0.537 (\pm 0.011)$	$0.534 (\pm 0.013)$	$0.543 (\pm 0.011)$	$0.525 (\pm 0.013)$	$0.513 (\pm 0.018)$	$0.535 (\pm 0.023)$	$0.546 (\pm 0.007)$	$0.563 (\pm 0.013)$
Avg.	$0.507 (\pm 0.022)$	$0.526 (\pm 0.011)$	$0.525 (\pm 0.014)$	$0.526 (\pm 0.013)$	$0.518 (\pm 0.012)$	$0.513 (\pm 0.022)$	$0.515 (\pm 0.028)$	$0.531 (\pm 0.011)$	$0.534 (\pm 0.022)$

方法	MiniDomainNet
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.509 (\pm 0.018)$	$0.526 (\pm 0.009)$	$0.524 (\pm 0.010)$	$0.523 (\pm 0.010)$	$0.518 (\pm 0.015)$	$0.511 (\pm 0.018)$	$0.511 (\pm 0.018)$	$0.531 (\pm 0.007)$	$0.537 (\pm 0.020)$
AdvSKM	$0.509 (\pm 0.015)$	$0.516 (\pm 0.011$	$0.514 (\pm 0.014)$	$0.514 (\pm 0.017)$	$0.514 (\pm 0.009)$	$0.515 (\pm 0.032)$	$0.512 (\pm 0.032)$	$0.516 (\pm 0.011)$	$0.522 (\pm 0.017)$
DIRT	$0.499 (\pm 0.022)$	$0.517 (\pm 0.006)$	$0.515 (\pm 0.009)$	$0.508 (\pm 0.006)$	$0.507 (\pm 0.018$	0.493(±0.028)	$0.498 (\pm 0.033)$	$0.519 (\pm 0.006$	$0.525 (\pm 0.020)$
DDC	$0.510 (\pm 0.032)$	0.514(±0.012)	$0.512 (\pm 0.018)$	$0.511 (\pm 0.019$	$0.514 (\pm 0.015)$	$0.511 (\pm 0.028)$	$0.512 (\pm 0.043$	$0.516 (\pm 0.015$	$0.521 (\pm 0.019)$
CMD	$0.509 (\pm 0.019)$	$0.528 (\pm 0.011)$	$0.531 (\pm 0.016$	$0.531 (\pm {0.016}^{\circ}$	$0.522 (\pm 0.013$	$0.518 (\pm 0.025)$	$0.490 (\pm 0.037)$	$0.533 (\pm 0.009$	$0.533 (\pm 0.024)$
MMDA	$0.509 (\pm 0.021)$	0.524(±0.004)	$0.522 (\pm 0.005)$	$0.523 (\pm 0.007)$	$0.519 (\pm 0.012)$	$0.517 (\pm 0.008)$	$0.524 (\pm 0.006)$	$0.527 (\pm 0.009$	$0.531 (\pm 0.029)$
CoDATS	0.502(±0.026)	$0.535 (\pm 0.026)$	$0.533 (\pm 0.027)$	$0.536 (\pm 0.028$	$0.525 (\pm 0.015)$	$0.518 (\pm 0.031)$	0.524(±0.051)	$0.536 (\pm 0.025)$	$0.529 (\pm 0.042)$
深度珊瑚	$0.505 (\pm 0.022)$	$0.538 (\pm 0.009)$	$0.534 (\pm 0.013)$	$0.533 (\pm 0.012)$	$0.520 (\pm 0.010$	$0.518 (\pm 0.022)$	$0.524 (\pm 0.034$	$0.539 (\pm 0.012$	$0.535 (\pm 0.024)$
CDAN	$0.514 (\pm 0.028)$	$0.526 (\pm 0.013$	$0.524 (\pm 0.011)$	$0.525 (\pm 0.008)$	$0.517 (\pm 0.005)$	$0.513 (\pm 0.011)$	$0.516 (\pm 0.014)$	$0.532 (\pm 0.010$	$0.542 (\pm 0.017)$
DANN	$0.496 (\pm 0.023)$	$0.530 (\pm 0.011)$	$0.535 (\pm 0.011)$	$0.538 (\pm 0.005)$	$0.516 (\pm 0.011)$	$0.519 (\pm 0.015)$	$0.515 (\pm 0.013)$	$0.541 (\pm 0.006)$	$0.532 (\pm 0.014)$
DSAN	$0.509 (\pm 0.022)$	$0.537 (\pm 0.011)$	$0.534 (\pm 0.013)$	$0.543 (\pm 0.011)$	$0.525 (\pm 0.013)$	$0.513 (\pm 0.018)$	$0.535 (\pm 0.023)$	$0.546 (\pm 0.007)$	$0.563 (\pm 0.013)$
平均值	$0.507 (\pm 0.022)$	$0.526 (\pm 0.011)$	$0.525 (\pm 0.014)$	$0.526 (\pm 0.013)$	$0.518 (\pm 0.012)$	$0.513 (\pm 0.022)$	$0.515 (\pm 0.028)$	$0.531 (\pm 0.011)$	$0.534 (\pm 0.022)$

Method	Sleep-EDF
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.676 (\pm 0.036)$	$0.722 (\pm 0.017)$	$0.719 (\pm 0.023)$	$0.718 (\pm 0.021)$	$0.724 (\pm 0.032)$	$0.726 (\pm 0.046)$	$0.678 (\pm 0.035)$	$0.747 (\pm 0.025$	$0.715 (\pm 0.047)$
AdvSKM	$0.665 (\pm 0.058)$	$0.708 (\pm 0.023)$	$0.712 (\pm 0.027)$	$0.712 (\pm 0.032)$	$0.718 (\pm 0.030)$	$0.703 (\pm 0.069)$	$0.692 (\pm 0.038)$	$0.722 (\pm 0.025)$	$0.706 (\pm 0.054)$
DIRT	$0.656 (\pm 0.058)$	$0.743 (\pm 0.009)$	$0.745 (\pm 0.012)$	$0.748 (\pm 0.019)$	$0.742 (\pm 0.031)$	$0.679 (\pm {0.038}^{\circ}$	$0.686 (\pm 0.066)$	$0.749 (\pm 0.010)$	$0.728 (\pm {0.037}^{\circ})$
DDC	$0.646 (\pm 0.035)$	$0.717 (\pm 0.029)$	0.721(±0.037	$0.712 (\pm 0.031)$	$0.695 (\pm 0.020)$	$0.694 (\pm 0.056)$	$0.666 (\pm {0.031}^{\circ})$	$0.724 (\pm 0.012)$	0.704(±0.031
CMD	$0.653 (\pm {0.057}^{\circ})$	0.740(±0.022	$0.736 (\pm 0.016)$	$0.723 (\pm 0.020)$	$0.709 (\pm 0.015)$	$0.716 (\pm 0.052)$	$0.640 (\pm 0.068)$	$0.729 (\pm {0.018}^{\circ}$	$0.725 (\pm 0.053)$
MMDA	$0.650 (\pm 0.051)$	$0.736 (\pm 0.014$	$0.727 (\pm 0.021)$	$0.723 (\pm 0.018)$	$0.714 (\pm 0.028$	$0.704 (\pm 0.033)$	$0.660 (\pm 0.034)$	$0.745 (\pm 0.031$	$0.715 (\pm 0.042)$
CoDATS	$0.672 (\pm 0.084)$	$0.738 (\pm 0.029)$	$0.739 (\pm 0.036$	$0.736 (\pm 0.030)$	$0.723 (\pm 0.039)$	$0.683 (\pm 0.090)$	$0.690 (\pm {0.107}^{\circ})$	0.744(±0.021	0.715(±0.045
Deep-Coral	$0.643 (\pm 0.049)$	$0.716 (\pm 0.018)$	0.717(±0.028	$0.712 (\pm 0.027)$	$0.694 \hat{i} \pm {0.032}^{\hat{ı}}$	$0.700 (\pm 0.053)$	$0.675 (\pm 0.077)$	$0.713 (\pm 0.021)$	$0.702 (\pm {0.070}^{\circ})$
CDAN	$0.652 l \pm {0.056}^{l}$	$0.732 (\pm 0.016)$	$0.739 (\pm 0.024)$	$0.739 \hat{i} \pm 0.01 \hat{8}$	$0.728 \dot{} \dot{} \dot{} \pm {0.029}^{\dot{}}$	$0.697 (\pm 0.031)$	$0.642 (\pm 0.065)$	$0.748 (\pm {0.019}^{\circ})$	$0.713 (\pm 0.045)$
DANN	$0.641 (\pm 0.047)$	$0.722 (\pm 0.017)$	$0.723 (\pm 0.026)$	$0.721 (\pm 0.025)$	$0.714 (\pm 0.024)$	$0.687 (\pm 0.034)$	$0.644 (\pm 0.046)$	$0.724 (\pm 0.018$	$0.710 (\pm 0.035)$
DSAN	$0.653 (\pm 0.060)$	0.748(±0.008)	$0.740 (\pm 0.016)$	$0.732 (\pm 0.016)$	$0.728 (\pm 0.026)$	$0.712 (\pm 0.070)$	$0.589 (\pm 0.063)$	$0.757 (\pm 0.016)$	$0.700 (\pm 0.033)$
Avg.	$0.655 (\pm 0.054)$	$0.729 (\pm 0.018)$	$0.729 (\pm 0.024)$	$0.725 (\pm 0.023)$	$0.717 (\pm 0.028)$	$0.700 (\pm 0.052)$	$0.660 (\pm 0.057)$	$0.737 (\pm 0.020)$	$0.712 (\pm 0.045)$

方法	睡眠-EDF
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.676 (\pm 0.036)$	$0.722 (\pm 0.017)$	$0.719 (\pm 0.023)$	$0.718 (\pm 0.021)$	$0.724 (\pm 0.032)$	$0.726 (\pm 0.046)$	$0.678 (\pm 0.035)$	$0.747 (\pm 0.025$	$0.715 (\pm 0.047)$
AdvSKM	$0.665 (\pm 0.058)$	$0.708 (\pm 0.023)$	$0.712 (\pm 0.027)$	$0.712 (\pm 0.032)$	$0.718 (\pm 0.030)$	$0.703 (\pm 0.069)$	$0.692 (\pm 0.038)$	$0.722 (\pm 0.025)$	$0.706 (\pm 0.054)$
DIRT	$0.656 (\pm 0.058)$	$0.743 (\pm 0.009)$	$0.745 (\pm 0.012)$	$0.748 (\pm 0.019)$	$0.742 (\pm 0.031)$	$0.679 (\pm {0.038}^{\circ}$	$0.686 (\pm 0.066)$	$0.749 (\pm 0.010)$	$0.728 (\pm {0.037}^{\circ})$
DDC	$0.646 (\pm 0.035)$	$0.717 (\pm 0.029)$	0.721(±0.037	$0.712 (\pm 0.031)$	$0.695 (\pm 0.020)$	$0.694 (\pm 0.056)$	$0.666 (\pm {0.031}^{\circ})$	$0.724 (\pm 0.012)$	0.704(±0.031
CMD	$0.653 (\pm {0.057}^{\circ})$	0.740(±0.022	$0.736 (\pm 0.016)$	$0.723 (\pm 0.020)$	$0.709 (\pm 0.015)$	$0.716 (\pm 0.052)$	$0.640 (\pm 0.068)$	$0.729 (\pm {0.018}^{\circ}$	$0.725 (\pm 0.053)$
MMDA	$0.650 (\pm 0.051)$	$0.736 (\pm 0.014$	$0.727 (\pm 0.021)$	$0.723 (\pm 0.018)$	$0.714 (\pm 0.028$	$0.704 (\pm 0.033)$	$0.660 (\pm 0.034)$	$0.745 (\pm 0.031$	$0.715 (\pm 0.042)$
CoDATS	$0.672 (\pm 0.084)$	$0.738 (\pm 0.029)$	$0.739 (\pm 0.036$	$0.736 (\pm 0.030)$	$0.723 (\pm 0.039)$	$0.683 (\pm 0.090)$	$0.690 (\pm {0.107}^{\circ})$	0.744(±0.021	0.715(±0.045
Deep-Coral	$0.643 (\pm 0.049)$	$0.716 (\pm 0.018)$	0.717(±0.028	$0.712 (\pm 0.027)$	$0.694 \hat{i} \pm {0.032}^{\hat{ı}}$	$0.700 (\pm 0.053)$	$0.675 (\pm 0.077)$	$0.713 (\pm 0.021)$	$0.702 (\pm {0.070}^{\circ})$
CDAN	$0.652 l \pm {0.056}^{l}$	$0.732 (\pm 0.016)$	$0.739 (\pm 0.024)$	$0.739 \hat{i} \pm 0.01 \hat{8}$	$0.728 \dot{} \dot{} \dot{} \pm {0.029}^{\dot{}}$	$0.697 (\pm 0.031)$	$0.642 (\pm 0.065)$	$0.748 (\pm {0.019}^{\circ})$	$0.713 (\pm 0.045)$
DANN	$0.641 (\pm 0.047)$	$0.722 (\pm 0.017)$	$0.723 (\pm 0.026)$	$0.721 (\pm 0.025)$	$0.714 (\pm 0.024)$	$0.687 (\pm 0.034)$	$0.644 (\pm 0.046)$	$0.724 (\pm 0.018$	$0.710 (\pm 0.035)$
DSAN	$0.653 (\pm 0.060)$	0.748(±0.008)	$0.740 (\pm 0.016)$	$0.732 (\pm 0.016)$	$0.728 (\pm 0.026)$	$0.712 (\pm 0.070)$	$0.589 (\pm 0.063)$	$0.757 (\pm 0.016)$	$0.700 (\pm 0.033)$
平均	$0.655 (\pm 0.054)$	$0.729 (\pm 0.018)$	$0.729 (\pm 0.024)$	$0.725 (\pm 0.023)$	$0.717 (\pm 0.028)$	$0.700 (\pm 0.052)$	$0.660 (\pm 0.057)$	$0.737 (\pm 0.020)$	$0.712 (\pm 0.045)$

Method	UCI-HAR
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.782 (\pm 0.078)$	$0.833 (\pm 0.020)$	$0.818 (\pm 0.023)$	$0.818 (\pm 0.022)$	$0.783 (\pm 0.040)$	$0.809 (\pm 0.095)$	$0.800 (\pm 0.098)$	$0.826 (\pm 0.010)$	$0.854 (\pm 0.039)$
AdvSKM	$0.724 (\pm 0.059)$	$0.791 (\pm 0.024)$	$0.800 (\pm 0.022)$	$0.810 (\pm 0.022)$	$0.768 (\pm 0.042)$	$0.707 (\pm 0.100)$	$0.711 (\pm 0.167)$	$0.800 (\pm 0.022)$	$0.811 (\pm 0.039)$
DIRT	$0.783 (\pm 0.044)$	0.912(±0.013	$0.907 (\pm 0.009)$	$0.890 (\pm 0.016)$	$0.756 (\pm 0.036$	$0.807 (\pm 0.107)$	$0.808 (\pm 0.112$	$0.900 (\pm 0.015)$	$0.928 (\pm 0.034)$
DDC	$0.790 (\pm 0.061)$	$0.806 (\pm 0.019)$	$0.807 (\pm 0.026)$	$0.810 (\pm 0.017$	$0.756 (\pm 0.108$	$0.724 (\pm 0.066)$	$0.734 (\pm 0.109)$	$0.804 (\pm 0.028$	$0.792 (\pm 0.013)$
CMD	$0.788 (\pm 0.058)$	$0.869 (\pm 0.012)$	$0.849 (\pm 0.014)$	$0.839 (\pm 0.023)$	$0.731 (\pm 0.066$	$0.804 (\pm 0.064)$	$0.812 (\pm 0.080)$	$0.842 (\pm 0.025)$	$0.888 (\pm 0.037)$
MMDA	$0.785 (\pm 0.018)$	$0.819 (\pm 0.022)$	$0.812 (\pm 0.028)$	$0.800 (\pm 0.032)$	$0.759 (\pm 0.085$	$0.773 (\pm 0.073)$	$0.767 (\pm 0.107)$	$0.807 (\pm 0.025$	$0.840 (\pm 0.055)$
CoDATS	$0.760 (\pm 0.037)$	$0.854 (\pm 0.022)$	$0.832 (\pm 0.027)$	$0.832 (\pm 0.006)$	$0.785 (\pm 0.057$	$0.801 (\pm 0.079)$	$0.794 (\pm 0.078)$	$0.846 (\pm 0.016)$	$0.867 (\pm 0.012)$
Deep-Coral	$0.790 (\pm {0.035}^{\circ})$	$0.810 (\pm 0.007$	$0.800 (\pm 0.022)$	$0.808 (\pm {0.030}^{\circ}$	$0.771 (\pm 0.023)$	$0.768 (\pm 0.044)$	$0.773 (\pm 0.087)$	$0.808 (\pm 0.016)$	$0.806 (\pm 0.022)$
CDAN	$0.756 (\pm 0.055)$	$0.842 (\pm 0.009)$	$0.843 (\pm 0.020)$	$0.840 (\pm 0.034)$	$0.802 (\pm 0.080)$	$0.781 (\pm 0.072)$	0.687(±0.068’	$0.846 (\pm 0.018)$	$0.853 (\pm 0.026)$
DANN	$0.756 (\pm 0.026)$	$0.858 (\pm 0.016)$	$0.856 (\pm 0.033)$	$0.856 (\pm 0.033)$	$0.800 (\pm 0.057)$	$0.763 (\pm 0.032)$	$0.780 (\pm 0.043)$	$0.849 (\pm 0.023)$	$0.847 (\pm 0.007)$
DSAN	$0.762 (\pm 0.032)$	$0.849 (\pm 0.023)$	$0.843 (\pm 0.033)$	$0.854 (\pm 0.025$	$0.749 (\pm 0.065)$	$0.775 (\pm 0.043)$	$0.744 (\pm 0.035)$	$0.858 (\pm 0.023)$	$0.865 (\pm 0.038)$
Avg.	$0.770 (\pm 0.046)$	$0.840 (\pm 0.017)$	$0.833 (\pm 0.023)$	$0.832 (\pm 0.024)$	$0.769 (\pm 0.060)$	$0.774 (\pm 0.070)$	$0.765 (\pm 0.090)$	$0.835 (\pm 0.020)$	$0.850 (\pm 0.029)$

方法	UCI-HAR
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.782 (\pm 0.078)$	$0.833 (\pm 0.020)$	$0.818 (\pm 0.023)$	$0.818 (\pm 0.022)$	$0.783 (\pm 0.040)$	$0.809 (\pm 0.095)$	$0.800 (\pm 0.098)$	$0.826 (\pm 0.010)$	$0.854 (\pm 0.039)$
AdvSKM	$0.724 (\pm 0.059)$	$0.791 (\pm 0.024)$	$0.800 (\pm 0.022)$	$0.810 (\pm 0.022)$	$0.768 (\pm 0.042)$	$0.707 (\pm 0.100)$	$0.711 (\pm 0.167)$	$0.800 (\pm 0.022)$	$0.811 (\pm 0.039)$
DIRT	$0.783 (\pm 0.044)$	0.912(±0.013	$0.907 (\pm 0.009)$	$0.890 (\pm 0.016)$	$0.756 (\pm 0.036$	$0.807 (\pm 0.107)$	$0.808 (\pm 0.112$	$0.900 (\pm 0.015)$	$0.928 (\pm 0.034)$
DDC	$0.790 (\pm 0.061)$	$0.806 (\pm 0.019)$	$0.807 (\pm 0.026)$	$0.810 (\pm 0.017$	$0.756 (\pm 0.108$	$0.724 (\pm 0.066)$	$0.734 (\pm 0.109)$	$0.804 (\pm 0.028$	$0.792 (\pm 0.013)$
CMD	$0.788 (\pm 0.058)$	$0.869 (\pm 0.012)$	$0.849 (\pm 0.014)$	$0.839 (\pm 0.023)$	$0.731 (\pm 0.066$	$0.804 (\pm 0.064)$	$0.812 (\pm 0.080)$	$0.842 (\pm 0.025)$	$0.888 (\pm 0.037)$
MMDA	$0.785 (\pm 0.018)$	$0.819 (\pm 0.022)$	$0.812 (\pm 0.028)$	$0.800 (\pm 0.032)$	$0.759 (\pm 0.085$	$0.773 (\pm 0.073)$	$0.767 (\pm 0.107)$	$0.807 (\pm 0.025$	$0.840 (\pm 0.055)$
CoDATS	$0.760 (\pm 0.037)$	$0.854 (\pm 0.022)$	$0.832 (\pm 0.027)$	$0.832 (\pm 0.006)$	$0.785 (\pm 0.057$	$0.801 (\pm 0.079)$	$0.794 (\pm 0.078)$	$0.846 (\pm 0.016)$	$0.867 (\pm 0.012)$
Deep-Coral	$0.790 (\pm {0.035}^{\circ})$	$0.810 (\pm 0.007$	$0.800 (\pm 0.022)$	$0.808 (\pm {0.030}^{\circ}$	$0.771 (\pm 0.023)$	$0.768 (\pm 0.044)$	$0.773 (\pm 0.087)$	$0.808 (\pm 0.016)$	$0.806 (\pm 0.022)$
CDAN	$0.756 (\pm 0.055)$	$0.842 (\pm 0.009)$	$0.843 (\pm 0.020)$	$0.840 (\pm 0.034)$	$0.802 (\pm 0.080)$	$0.781 (\pm 0.072)$	0.687(±0.068’	$0.846 (\pm 0.018)$	$0.853 (\pm 0.026)$
DANN	$0.756 (\pm 0.026)$	$0.858 (\pm 0.016)$	$0.856 (\pm 0.033)$	$0.856 (\pm 0.033)$	$0.800 (\pm 0.057)$	$0.763 (\pm 0.032)$	$0.780 (\pm 0.043)$	$0.849 (\pm 0.023)$	$0.847 (\pm 0.007)$
DSAN	$0.762 (\pm 0.032)$	$0.849 (\pm 0.023)$	$0.843 (\pm 0.033)$	$0.854 (\pm 0.025$	$0.749 (\pm 0.065)$	$0.775 (\pm 0.043)$	$0.744 (\pm 0.035)$	$0.858 (\pm 0.023)$	$0.865 (\pm 0.038)$
平均	$0.770 (\pm 0.046)$	$0.840 (\pm 0.017)$	$0.833 (\pm 0.023)$	$0.832 (\pm 0.024)$	$0.769 (\pm 0.060)$	$0.774 (\pm 0.070)$	$0.765 (\pm 0.090)$	$0.835 (\pm 0.020)$	$0.850 (\pm 0.029)$

Method	HHAR
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
HoMM	$0.739 (\pm 0.044)$	$0.757 (\pm 0.014)$	$0.759 (\pm 0.013)$	$0.759 (\pm 0.011)$	$0.700 (\pm 0.058)$	$0.720 (\pm 0.027)$	$0.733 (\pm 0.031)$	$0.759 (\pm 0.007)$	$0.764 (\pm 0.023)$
AdvSKM	$0.718 (\pm 0.042)$	$0.749 (\pm 0.027)$	$0.742 (\pm 0.032)$	$0.748 (\pm 0.034)$	$0.676 (\pm 0.046)$	$0.730 (\pm 0.051)$	$0.728 (\pm 0.051)$	$0.752 (\pm 0.031$	$0.749 (\pm 0.025)$
DIRT	$0.728 (\pm 0.026)$	$0.803 (\pm 0.011)$	$0.792 (\pm 0.016)$	$0.803 (\pm 0.017)$	$0.796 (\pm 0.066)$	$0.743 (\pm 0.028)$	0.739(±0.075	$0.816 (\pm 0.008)$	$0.820 (\pm 0.015)$
DDC	$0.716 (\pm 0.063)$	$0.748 (\pm 0.014)$	0.750(±0.009	$0.748 (\pm 0.007)$	$0.717 (\pm 0.075)$	$0.711 (\pm 0.048)$	0.705(±0.066	$0.748 (\pm 0.012)$	$0.729 (\pm 0.027)$
CMD	$0.748 (\pm 0.027)$	$0.760 (\pm 0.014)$	$0.764 (\pm 0.006)$	$0.767 (\pm 0.007)$	$0.737 (\pm 0.100)$	$0.775 (\pm 0.031$	$0.643 (\pm 0.031)$	$0.766 (\pm 0.016)$	$0.794 (\pm 0.023)$
MMDA	$0.738 (\pm 0.036)$	$0.783 (\pm 0.017)$	$0.781 (\pm 0.016)$	$0.780 (\pm 0.015)$	$0.698 (\pm 0.038)$	$0.719 (\pm 0.036)$	$0.731 (\pm {0.047}^{\circ})$	$0.780 (\pm 0.017)$	$0.785 (\pm 0.035)$
CoDATS	$0.710 (\pm 0.030)$	$0.766 (\pm 0.023)$	$0.772 (\pm 0.040)$	0.773(±0.050	$0.722 (\pm 0.064)$	$0.739 (\pm 0.028)$	$0.739 (\pm 0.040)$	0.812(±0.009)	$0.785 / \pm 0.039$
Deep-Coral	$0.745 (\pm 0.046)$	0.766(±0.012	$0.762 \hat{i} \pm 0.015 \hat{j}$	$0.766 (\pm 0.027)$	$0.681 (\pm 0.073)$	$0.754 (\pm 0.054)$	$0.758 (\pm 0.244)$	$0.764 (\pm 0.006)$	$0.776 (\pm 0.023)$
CDAN	$0.728 (\pm 0.039)$	$0.762 l \pm 0.012)$	$0.758 (\pm 0.017)$	$0.764 (\pm 0.016)$	0.765(±0.063	$0.774 (\pm 0.035)$	$0.775 (\pm 0.036)$	0.816(±0.011)	$0.790 (\pm 0.038)$
DANN	$0.757 (\pm 0.057)$	$0.779 (\pm 0.012)$	$0.774 (\pm 0.009)$	$0.773 (\pm 0.011)$	$0.722 (\pm 0.103)$	$0.798 (\pm 0.041)$	$0.793 (\pm 0.045)$	$0.818 (\pm 0.009)$	$0.807 (\pm 0.020)$
DSAN	$0.721 (\pm 0.053)$	$0.803 (\pm 0.010)$	$0.797 (\pm 0.014)$	$0.802 (\pm 0.007)$	$0.724 (\pm 0.065)$	$0.741 (\pm 0.033)$	$0.596 (\pm 0.031)$	$0.825 (\pm 0.008)$	$0.826 (\pm 0.046)$
Avg.	$0.732 (\pm 0.042)$	$0.771 (\pm 0.015)$	$0.768 (\pm 0.017)$	$0.771 (\pm 0.018)$	$0.722 (\pm 0.068)$	$0.746 (\pm 0.037)$	$0.722 (\pm 0.063)$	$0.787 (\pm 0.012)$	$0.784 (\pm 0.028)$

方法	HHAR
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
HoMM	$0.739 (\pm 0.044)$	$0.757 (\pm 0.014)$	$0.759 (\pm 0.013)$	$0.759 (\pm 0.011)$	$0.700 (\pm 0.058)$	$0.720 (\pm 0.027)$	$0.733 (\pm 0.031)$	$0.759 (\pm 0.007)$	$0.764 (\pm 0.023)$
AdvSKM	$0.718 (\pm 0.042)$	$0.749 (\pm 0.027)$	$0.742 (\pm 0.032)$	$0.748 (\pm 0.034)$	$0.676 (\pm 0.046)$	$0.730 (\pm 0.051)$	$0.728 (\pm 0.051)$	$0.752 (\pm 0.031$	$0.749 (\pm 0.025)$
DIRT	$0.728 (\pm 0.026)$	$0.803 (\pm 0.011)$	$0.792 (\pm 0.016)$	$0.803 (\pm 0.017)$	$0.796 (\pm 0.066)$	$0.743 (\pm 0.028)$	0.739(±0.075	$0.816 (\pm 0.008)$	$0.820 (\pm 0.015)$
DDC	$0.716 (\pm 0.063)$	$0.748 (\pm 0.014)$	0.750(±0.009	$0.748 (\pm 0.007)$	$0.717 (\pm 0.075)$	$0.711 (\pm 0.048)$	0.705(±0.066	$0.748 (\pm 0.012)$	$0.729 (\pm 0.027)$
CMD	$0.748 (\pm 0.027)$	$0.760 (\pm 0.014)$	$0.764 (\pm 0.006)$	$0.767 (\pm 0.007)$	$0.737 (\pm 0.100)$	$0.775 (\pm 0.031$	$0.643 (\pm 0.031)$	$0.766 (\pm 0.016)$	$0.794 (\pm 0.023)$
MMDA	$0.738 (\pm 0.036)$	$0.783 (\pm 0.017)$	$0.781 (\pm 0.016)$	$0.780 (\pm 0.015)$	$0.698 (\pm 0.038)$	$0.719 (\pm 0.036)$	$0.731 (\pm {0.047}^{\circ})$	$0.780 (\pm 0.017)$	$0.785 (\pm 0.035)$
CoDATS	$0.710 (\pm 0.030)$	$0.766 (\pm 0.023)$	$0.772 (\pm 0.040)$	0.773(±0.050	$0.722 (\pm 0.064)$	$0.739 (\pm 0.028)$	$0.739 (\pm 0.040)$	0.812(±0.009)	$0.785 / \pm 0.039$
深度珊瑚	$0.745 (\pm 0.046)$	0.766(±0.012)	$0.762 \hat{i} \pm 0.015 \hat{j}$	$0.766 (\pm 0.027)$	$0.681 (\pm 0.073)$	$0.754 (\pm 0.054)$	$0.758 (\pm 0.244)$	$0.764 (\pm 0.006)$	$0.776 (\pm 0.023)$
CDAN	$0.728 (\pm 0.039)$	$0.762 l \pm 0.012)$	$0.758 (\pm 0.017)$	$0.764 (\pm 0.016)$	0.765(±0.063)	$0.774 (\pm 0.035)$	$0.775 (\pm 0.036)$	0.816(±0.011)	$0.790 (\pm 0.038)$
DANN	$0.757 (\pm 0.057)$	$0.779 (\pm 0.012)$	$0.774 (\pm 0.009)$	$0.773 (\pm 0.011)$	$0.722 (\pm 0.103)$	$0.798 (\pm 0.041)$	$0.793 (\pm 0.045)$	$0.818 (\pm 0.009)$	$0.807 (\pm 0.020)$
DSAN	$0.721 (\pm 0.053)$	$0.803 (\pm 0.010)$	$0.797 (\pm 0.014)$	$0.802 (\pm 0.007)$	$0.724 (\pm 0.065)$	$0.741 (\pm 0.033)$	$0.596 (\pm 0.031)$	$0.825 (\pm 0.008)$	$0.826 (\pm 0.046)$
平均	$0.732 (\pm 0.042)$	$0.771 (\pm 0.015)$	$0.768 (\pm 0.017)$	$0.771 (\pm 0.018)$	$0.722 (\pm 0.068)$	$0.746 (\pm 0.037)$	$0.722 (\pm 0.063)$	$0.787 (\pm 0.012)$	$0.784 (\pm 0.028)$

Method	WISDM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.753 (\pm 0.054)$	$0.741 (\pm 0.026)$	$0.738 (\pm 0.031)$	$0.739 (\pm 0.047)$	0.775\{±0.062	$0.753 (\pm 0.054)$	$0.740 (\pm 0.054)$	$0.728 (\pm 0.021)$	$0.774 (\pm 0.037)$
AdvSKM	$0.747 (\pm 0.050)$	$0.771 (\pm 0.043)$	0.781 $(\pm 0.055)$	$0.779 (\pm 0.035)$	$0.742 (\pm 0.062)$	$0.747 (\pm 0.050)$	$0.747 (\pm {0.135}^{\circ})$	$0.777 (\pm 0.031)$	$0.779 (\pm 0.041)$
DIRT	$0.738 (\pm 0.038)$	$0.792 (\pm 0.015)$	0.797(±0.024	$0.797 (\pm 0.037)$	$0.756 (\pm 0.071)$	$0.738 (\pm 0.059)$	$0.797 (\pm 0.059)$	$0.798 (\pm 0.018)$	$0.816 (\pm 0.063)$
DDC	$0.741 (\pm 0.071)$	$0.780 (\pm 0.032)$	$0.779 (\pm 0.052)$	$0.787 (\pm 0.049)$	$0.737 (\pm 0.071)$	$0.741 (\pm 0.076)$	$0.741 (\pm 0.063)$	$0.782 (\pm 0.038)$	$0.770 (\pm 0.060)$
CMD	$0.710 (\pm 0.088)$	0.772(±0.021	$0.765 (\pm 0.032)$	$0.767 (\pm 0.040)$	$0.728 (\pm 0.092)$	$0.713 (\pm 0.084)$	$0.686 (\pm 0.113)$	$0.773 (\pm 0.032)$	$0.742 (\pm 0.071)$
MMDA	$0.759 (\pm 0.047)$	0.789(±0.017	$0.772 (\pm 0.030)$	$0.745 (\pm 0.035)$	$0.754 (\pm 0.050)$	$0.759 (\pm 0.047)$	$0.750 (\pm 0.047)$	$0.790 (\pm 0.018)$	$0.775 (\pm 0.030)$
CoDATS	$0.711 (\pm 0.039)$	$0.775 (\pm 0.018$	$0.757 (\pm 0.027)$	$0.751 (\pm 0.020)$	$0.682 (\pm 0.057)$	$0.709 (\pm 0.039)$	$0.735 (\pm 0.054)$	$0.764 (\pm 0.015)$	$0.770 (\pm 0.019)$
Deep-Coral	$0.694 (\pm 0.030)$	$0.717 (\pm {0.041}^{\circ})$	0.723(±0.037	$0.713 (\pm 0.035)$	$0.664 (\pm 0.055)$	$0.694 (\pm 0.030)$	$0.670 (\pm 0.149)$	$0.723 (\pm 0.026)$	$0.736 (\pm 0.044)$
CDAN	$0.760 (\pm 0.057)$	$0.762 \dot{ℓ} \pm {0.048}^{\circ}$	$0.762 (\pm 0.046)$	0.781(±0.051	$0.768 (\pm 0.074)$	$0.760 (\pm 0.057)$	$0.750 (\pm 0.091)$	$0.765 (\pm 0.040)$	$0.779 (\pm 0.049)$
DANN	$0.724 (\pm 0.042)$	$0.789 \dot{ℓ} \pm {0.018}^{\dot{ℓ}}$	0.802(±0.028	$0.796 (\pm 0.036)$	$0.745 (\pm 0.026)$	$0.720 (\pm 0.042)$	$0.702 (\pm {0.053}^{\circ})$	$0.778 (\pm {0.019}^{\circ})$	$0.765 (\pm 0.044)$
DSAN	$0.759 (\pm 0.030)$	$0.765 (\pm 0.024)$	$0.769 (\pm 0.034)$	$0.756 (\pm 0.025)$	$0.757 (\pm 0.068)$	$0.759 (\pm 0.030)$	$0.663 (\pm 0.025)$	$0.722 (\pm 0.013)$	$0.779 (\pm 0.044)$
Avg.	$0.736 (\pm 0.050)$	$0.768 (\pm 0.027)$	$0.768 (\pm 0.036)$	$0.765 (\pm 0.037)$	$0.737 (\pm 0.062)$	$0.736 (\pm 0.052)$	$0.726 (\pm 0.077)$	$0.764 (\pm 0.025)$	$0.771 (\pm 0.046)$

方法	WISDM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.753 (\pm 0.054)$	$0.741 (\pm 0.026)$	$0.738 (\pm 0.031)$	$0.739 (\pm 0.047)$	0.775{±0.062	$0.753 (\pm 0.054)$	$0.740 (\pm 0.054)$	$0.728 (\pm 0.021)$	$0.774 (\pm 0.037)$
AdvSKM	$0.747 (\pm 0.050)$	$0.771 (\pm 0.043)$	0.781 $(\pm 0.055)$	$0.779 (\pm 0.035)$	$0.742 (\pm 0.062)$	$0.747 (\pm 0.050)$	$0.747 (\pm {0.135}^{\circ})$	$0.777 (\pm 0.031)$	$0.779 (\pm 0.041)$
DIRT	$0.738 (\pm 0.038)$	$0.792 (\pm 0.015)$	0.797(±0.024	$0.797 (\pm 0.037)$	$0.756 (\pm 0.071)$	$0.738 (\pm 0.059)$	$0.797 (\pm 0.059)$	$0.798 (\pm 0.018)$	$0.816 (\pm 0.063)$
DDC	$0.741 (\pm 0.071)$	$0.780 (\pm 0.032)$	$0.779 (\pm 0.052)$	$0.787 (\pm 0.049)$	$0.737 (\pm 0.071)$	$0.741 (\pm 0.076)$	$0.741 (\pm 0.063)$	$0.782 (\pm 0.038)$	$0.770 (\pm 0.060)$
CMD	$0.710 (\pm 0.088)$	0.772(±0.021	$0.765 (\pm 0.032)$	$0.767 (\pm 0.040)$	$0.728 (\pm 0.092)$	$0.713 (\pm 0.084)$	$0.686 (\pm 0.113)$	$0.773 (\pm 0.032)$	$0.742 (\pm 0.071)$
MMDA	$0.759 (\pm 0.047)$	0.789(±0.017	$0.772 (\pm 0.030)$	$0.745 (\pm 0.035)$	$0.754 (\pm 0.050)$	$0.759 (\pm 0.047)$	$0.750 (\pm 0.047)$	$0.790 (\pm 0.018)$	$0.775 (\pm 0.030)$
CoDATS	$0.711 (\pm 0.039)$	$0.775 (\pm 0.018$	$0.757 (\pm 0.027)$	$0.751 (\pm 0.020)$	$0.682 (\pm 0.057)$	$0.709 (\pm 0.039)$	$0.735 (\pm 0.054)$	$0.764 (\pm 0.015)$	$0.770 (\pm 0.019)$
深度珊瑚	$0.694 (\pm 0.030)$	$0.717 (\pm {0.041}^{\circ})$	0.723(±0.037)	$0.713 (\pm 0.035)$	$0.664 (\pm 0.055)$	$0.694 (\pm 0.030)$	$0.670 (\pm 0.149)$	$0.723 (\pm 0.026)$	$0.736 (\pm 0.044)$
CDAN	$0.760 (\pm 0.057)$	$0.762 \dot{ℓ} \pm {0.048}^{\circ}$	$0.762 (\pm 0.046)$	0.781(±0.051)	$0.768 (\pm 0.074)$	$0.760 (\pm 0.057)$	$0.750 (\pm 0.091)$	$0.765 (\pm 0.040)$	$0.779 (\pm 0.049)$
DANN	$0.724 (\pm 0.042)$	$0.789 \dot{ℓ} \pm {0.018}^{\dot{ℓ}}$	0.802(±0.028)	$0.796 (\pm 0.036)$	$0.745 (\pm 0.026)$	$0.720 (\pm 0.042)$	$0.702 (\pm {0.053}^{\circ})$	$0.778 (\pm {0.019}^{\circ})$	$0.765 (\pm 0.044)$
DSAN	$0.759 (\pm 0.030)$	$0.765 (\pm 0.024)$	$0.769 (\pm 0.034)$	$0.756 (\pm 0.025)$	$0.757 (\pm 0.068)$	$0.759 (\pm 0.030)$	$0.663 (\pm 0.025)$	$0.722 (\pm 0.013)$	$0.779 (\pm 0.044)$
平均值	$0.736 (\pm 0.050)$	$0.768 (\pm 0.027)$	$0.768 (\pm 0.036)$	$0.765 (\pm 0.037)$	$0.737 (\pm 0.062)$	$0.736 (\pm 0.052)$	$0.726 (\pm 0.077)$	$0.764 (\pm 0.025)$	$0.771 (\pm 0.046)$

Task	HoMM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.778 (\pm 0.01)$	$0.793 (\pm 0.008)$	$0.793 (\pm 0.008)$	$0.795 (\pm 0.002)$	$0.795 (\pm 0.008)$	$0.782 (\pm 0.021)$	$0.782 (\pm 0.021)$	$0.794 (\pm 0.007)$	$0.789 (\pm 0.008)$
$B \to E$	$0.745 (\pm 0.014)$	$0.759 (\pm 0.013)$	$0.759 (\pm 0.013)$	$0.761 (\pm 0.012)$	$0.763 (\pm 0.012$	$0.747 (\pm 0.005)$	$0.749 (\pm 0.025)$	$0.761 (\pm 0.012)$	$0.758 (\pm 0.012)$
$B \to K$	$0.77 (\pm 0.018)$	$0.782 (\pm 0.018$	$0.782 (\pm 0.018)$	$0.782 (\pm 0.017)$	$0.779 (\pm 0.013)$	$0.761 (\pm 0.018$	$0.767 (\pm 0.003)$	$0.781 (\pm 0.017)$	$0.776 (\pm 0.018)$
$D \to B$	$0.785 (\pm 0.006)$	$0.791 (\pm 0.004$	$0.791 (\pm 0.004)$	$0.79 (\pm 0.002)$	$0.788 (\pm 0.008)$	$0.778 (\pm 0.002$	$0.777 (\pm 0.015)$	$0.79 (\pm 0.003)$	$0.787 (\pm 0.003)$
$D \to E$	$0.772 (\pm 0.012)$	$0.781 (\pm 0.003)$	$0.781 (\pm 0.003)$	$0.779 (\pm 0.004)$	$0.778 (\pm 0.004)$	$0.764 (\pm 0.017)$	$0.769 (\pm 0.009$	$0.781 (\pm 0.004)$	$0.774 (\pm 0.003)$
$D \to K$	$0.783 (\pm 0.015)$	$0.791 (\pm 0.016)$	$0.791 (\pm 0.016)$	$0.791 (\pm 0.016)$	$0.791 (\pm 0.017)$	$0.781 ⟨ \pm 0.01 ⟩$	$0.782 (\pm 0.01)$	$0.789 (\pm 0.016$	$0.786 (\pm 0.017)$
$E \to B$	$0.702 (\pm 0.026)$	$0.7 (\pm 0.02)$	$0.7 (\pm 0.02)$	$0.701 (\pm 0.019$	$0.699 (\pm 0.021)$	$0.69 (\pm 0.011)$	$0.692 (\pm 0.014)$	$0.698 (\pm 0.011$	$0.703 (\pm 0.03)$
$E \to D$	$0.733 (\pm 0.003)$	$0.733 (\pm 0.006)$	$0.733 (\pm 0.006)$	0.736(±0.011	$0.733 (\pm 0.008)$	$0.728 (\pm 0.011)$	0.719(±0.005)	$0.735 (\pm 0.011)$	$0.735 (\pm 0.011)$
$E \to K$	$0.862 (\pm 0.009)$	$0.874 (\pm 0.008)$	$0.874 (\pm 0.008)$	$0.876 (\pm 0.009$	$0.873 (\pm 0.008)$	$0.856 (\pm 0.01)$	$0.858 (\pm 0.012)$	$0.873 (\pm 0.009)$	$0.863 (\pm 0.009)$
$K \to B$	$0.715 (\pm 0.008)$	0.721(±0.004	$0.721 (\pm 0.004)$	$0.719 (\pm 0.004)$	$0.721 (\pm 0.01)$	$0.714 (\pm 0.007)$	$0.708 (\pm 0.003)$	$0.722 (\pm 0.007$	$0.717 (\pm 0.007)$
$K \to D$	$0.745 (\pm 0.011)$	$0.752 (\pm 0.01)$	$0.752 (\pm 0.01)$	$0.747 (\pm 0.013)$	$0.755 (\pm 0.008$	$0.735 (\pm 0.018)$	$0.742 (\pm 0.022)$	$0.749 (\pm 0.006)$	$0.745 (\pm 0.008)$
$K \to E$	$0.844 (\pm 0.009)$	$0.854 (\pm 0.008)$	$0.854 (\pm 0.008)$	$0.855 (\pm 0.005)$	$0.857 (\pm 0.009$	$0.845 (\pm 0.005)$	$0.844 (\pm 0.006)$	$0.856 (\pm 0.006)$	$0.849 (\pm 0.009)$
Avg.	$0.769 (\pm 0.012)$	$0.778 (\pm 0.01)$	$0.778 (\pm 0.01)$	$0.778 (\pm 0.01)$	$0.777 (\pm 0.011)$	$0.765 (\pm 0.011)$	$0.766 (\pm 0.012)$	$0.778 (\pm 0.009)$	$0.774 (\pm 0.011)$

任务	HoMM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$B \to D$	$0.778 (\pm 0.01)$	$0.793 (\pm 0.008)$	$0.793 (\pm 0.008)$	$0.795 (\pm 0.002)$	$0.795 (\pm 0.008)$	$0.782 (\pm 0.021)$	$0.782 (\pm 0.021)$	$0.794 (\pm 0.007)$	$0.789 (\pm 0.008)$
$B \to E$	$0.745 (\pm 0.014)$	$0.759 (\pm 0.013)$	$0.759 (\pm 0.013)$	$0.761 (\pm 0.012)$	$0.763 (\pm 0.012$	$0.747 (\pm 0.005)$	$0.749 (\pm 0.025)$	$0.761 (\pm 0.012)$	$0.758 (\pm 0.012)$
$B \to K$	$0.77 (\pm 0.018)$	$0.782 (\pm 0.018$	$0.782 (\pm 0.018)$	$0.782 (\pm 0.017)$	$0.779 (\pm 0.013)$	$0.761 (\pm 0.018$	$0.767 (\pm 0.003)$	$0.781 (\pm 0.017)$	$0.776 (\pm 0.018)$
$D \to B$	$0.785 (\pm 0.006)$	$0.791 (\pm 0.004$	$0.791 (\pm 0.004)$	$0.79 (\pm 0.002)$	$0.788 (\pm 0.008)$	$0.778 (\pm 0.002$	$0.777 (\pm 0.015)$	$0.79 (\pm 0.003)$	$0.787 (\pm 0.003)$
$D \to E$	$0.772 (\pm 0.012)$	$0.781 (\pm 0.003)$	$0.781 (\pm 0.003)$	$0.779 (\pm 0.004)$	$0.778 (\pm 0.004)$	$0.764 (\pm 0.017)$	$0.769 (\pm 0.009$	$0.781 (\pm 0.004)$	$0.774 (\pm 0.003)$
$D \to K$	$0.783 (\pm 0.015)$	$0.791 (\pm 0.016)$	$0.791 (\pm 0.016)$	$0.791 (\pm 0.016)$	$0.791 (\pm 0.017)$	$0.781 ⟨ \pm 0.01 ⟩$	$0.782 (\pm 0.01)$	$0.789 (\pm 0.016$	$0.786 (\pm 0.017)$
$E \to B$	$0.702 (\pm 0.026)$	$0.7 (\pm 0.02)$	$0.7 (\pm 0.02)$	$0.701 (\pm 0.019$	$0.699 (\pm 0.021)$	$0.69 (\pm 0.011)$	$0.692 (\pm 0.014)$	$0.698 (\pm 0.011$	$0.703 (\pm 0.03)$
$E \to D$	$0.733 (\pm 0.003)$	$0.733 (\pm 0.006)$	$0.733 (\pm 0.006)$	0.736(±0.011	$0.733 (\pm 0.008)$	$0.728 (\pm 0.011)$	0.719(±0.005)	$0.735 (\pm 0.011)$	$0.735 (\pm 0.011)$
$E \to K$	$0.862 (\pm 0.009)$	$0.874 (\pm 0.008)$	$0.874 (\pm 0.008)$	$0.876 (\pm 0.009$	$0.873 (\pm 0.008)$	$0.856 (\pm 0.01)$	$0.858 (\pm 0.012)$	$0.873 (\pm 0.009)$	$0.863 (\pm 0.009)$
$K \to B$	$0.715 (\pm 0.008)$	0.721(±0.004)	$0.721 (\pm 0.004)$	$0.719 (\pm 0.004)$	$0.721 (\pm 0.01)$	$0.714 (\pm 0.007)$	$0.708 (\pm 0.003)$	$0.722 (\pm 0.007$	$0.717 (\pm 0.007)$
$K \to D$	$0.745 (\pm 0.011)$	$0.752 (\pm 0.01)$	$0.752 (\pm 0.01)$	$0.747 (\pm 0.013)$	$0.755 (\pm 0.008$	$0.735 (\pm 0.018)$	$0.742 (\pm 0.022)$	$0.749 (\pm 0.006)$	$0.745 (\pm 0.008)$
$K \to E$	$0.844 (\pm 0.009)$	$0.854 (\pm 0.008)$	$0.854 (\pm 0.008)$	$0.855 (\pm 0.005)$	$0.857 (\pm 0.009$	$0.845 (\pm 0.005)$	$0.844 (\pm 0.006)$	$0.856 (\pm 0.006)$	$0.849 (\pm 0.009)$
平均值	$0.769 (\pm 0.012)$	$0.778 (\pm 0.01)$	$0.778 (\pm 0.01)$	$0.778 (\pm 0.01)$	$0.777 (\pm 0.011)$	$0.765 (\pm 0.011)$	$0.766 (\pm 0.012)$	$0.778 (\pm 0.009)$	$0.774 (\pm 0.011)$

Task	AdvSKM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.784 (\pm 0.004)$	$0.79 (\pm 0.003)$	$0.79 (\pm 0.003)$	$0.789 (\pm 0.003)$	$0.79 (\pm 0.003)$	$0.777 (\pm 0.008)$	$0.778 (\pm 0.008)$	$0.794 (\pm 0.005)$	$0.79 (\pm 0.005)$
$B \to E$	$0.754 (\pm 0.009)$	$0.761 (\pm 0.014)$	$0.761 (\pm 0.014)$	0.763(±0.012	$0.762 (\pm 0.011)$	$0.754 (\pm 0.003)$	$0.751 (\pm 0.007)$	$0.761 (\pm 0.012)$	$0.756 (\pm 0.012)$
$B \to K$	$0.769 (\pm 0.021)$	$0.784 (\pm 0.019)$	$0.784 (\pm 0.019)$	$0.782 (\pm 0.021)$	$0.78 (\pm 0.016)$	$0.771 (\pm 0.012)$	$0.764 (\pm 0.016)$	$0.782 (\pm 0.017)$	$0.774 (\pm 0.019$
$D \to B$	$0.779 (\pm 0.004)$	$0.795 (\pm 0.007)$	$0.795 (\pm 0.007)$	$0.794 (\pm 0.007)$	$0.792 (\pm 0.003)$	$0.785 (\pm 0.007)$	$0.78 (\pm 0.011)$	$0.796 (\pm 0.009)$	$0.789 (\pm {0.009}^{\circ}$
$D \to E$	$0.766 (\pm 0.013)$	$0.785 (\pm 0.006)$	$0.785 (\pm 0.006)$	$0.784 (\pm 0.008)$	$0.777 (\pm 0.005)$	$0.773 (\pm 0.005)$	$0.769 (\pm 0.009)$	$0.785 (\pm 0.005)$	$0.776 (\pm 0.002)$
$D \to K$	$0.777 (\pm 0.015)$	$0.79 (\pm 0.015)$	$0.79 (\pm 0.015)$	$0.79 (\pm 0.015)$	$0.791 (\pm 0.013$	$0.775 (\pm 0.014)$	$0.775 (\pm 0.014)$	$0.789 (\pm 0.013)$	$0.786 (\pm 0.013)$
$E \to B$	$0.688 (\pm 0.015)$	$0.709 (\pm 0.022$	$0.709 (\pm 0.022)$	$0.705 (\pm 0.024)$	$0.709 (\pm 0.021$	$0.704 (\pm {0.022}^{\circ}$	$0.695 (\pm 0.022)$	$0.712 (\pm 0.021)$	$0.707 (\pm 0.021$
$E \to D$	$0.714 (\pm 0.005)$	$0.734 (\pm 0.002)$	$0.734 (\pm 0.002)$	$0.737 (\pm 0.005$	$0.733 (\pm 0.008)$	$0.725 (\pm 0.004)$	$0.725 (\pm 0.004)$	$0.732 (\pm 0.006)$	$0.737 (\pm 0.011)$
$E \to K$	$0.858 (\pm 0.004)$	$0.874 (\pm 0.01)$	$0.874 (\pm 0.01)$	$0.874 (\pm 0.012)$	$0.876 (\pm 0.01)$	$0.86 (\pm 0.012)$	$0.86 (\pm 0.012)$	$0.875 (\pm 0.011)$	$0.865 (\pm 0.01)$
$K \to B$	$0.713 (\pm 0.002)$	$0.723 (\pm 0.006)$	$0.723 (\pm 0.006)$	$0.724 (\pm 0.009)$	$0.72 (\pm 0.004)$	$0.708 (\pm 0.014)$	$0.705 (\pm 0.011)$	$0.726 (\pm 0.006)$	$0.718 (\pm 0.006$
$K \to D$	$0.741 (\pm 0.01)$	$0.752 (\pm 0.012)$	$0.752 (\pm 0.012)$	$0.751 (\pm 0.013)$	$0.752 (\pm {0.017}^{\circ})$	$0.75 (\pm 0.019)$	$0.742 (\pm 0.008)$	$0.749 (\pm 0.012)$	$0.756 (\pm 0.014$
$K \to E$	$0.848 (\pm 0.008)$	$0.857 (\pm 0.007)$	$0.857 (\pm 0.007)$	$0.858 (\pm 0.008)$	$0.856 (\pm 0.008)$	$0.846 (\pm 0.012)$	$0.847 (\pm 0.007)$	$0.858 (\pm 0.008)$	$0.85 (\pm 0.008)$
Avg.	$0.766 (\pm 0.009)$	$0.779 (\pm 0.01)$	$0.779 (\pm 0.01)$	$0.779 (\pm 0.011)$	$0.778 (\pm 0.01)$	$0.769 (\pm 0.011)$	$0.766 (\pm 0.011)$	$0.78 (\pm 0.01)$	$0.775 (\pm 0.011)$

任务	AdvSKM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$B \to D$	$0.784 (\pm 0.004)$	$0.79 (\pm 0.003)$	$0.79 (\pm 0.003)$	$0.789 (\pm 0.003)$	$0.79 (\pm 0.003)$	$0.777 (\pm 0.008)$	$0.778 (\pm 0.008)$	$0.794 (\pm 0.005)$	$0.79 (\pm 0.005)$
$B \to E$	$0.754 (\pm 0.009)$	$0.761 (\pm 0.014)$	$0.761 (\pm 0.014)$	0.763(±0.012	$0.762 (\pm 0.011)$	$0.754 (\pm 0.003)$	$0.751 (\pm 0.007)$	$0.761 (\pm 0.012)$	$0.756 (\pm 0.012)$
$B \to K$	$0.769 (\pm 0.021)$	$0.784 (\pm 0.019)$	$0.784 (\pm 0.019)$	$0.782 (\pm 0.021)$	$0.78 (\pm 0.016)$	$0.771 (\pm 0.012)$	$0.764 (\pm 0.016)$	$0.782 (\pm 0.017)$	$0.774 (\pm 0.019$
$D \to B$	$0.779 (\pm 0.004)$	$0.795 (\pm 0.007)$	$0.795 (\pm 0.007)$	$0.794 (\pm 0.007)$	$0.792 (\pm 0.003)$	$0.785 (\pm 0.007)$	$0.78 (\pm 0.011)$	$0.796 (\pm 0.009)$	$0.789 (\pm {0.009}^{\circ}$
$D \to E$	$0.766 (\pm 0.013)$	$0.785 (\pm 0.006)$	$0.785 (\pm 0.006)$	$0.784 (\pm 0.008)$	$0.777 (\pm 0.005)$	$0.773 (\pm 0.005)$	$0.769 (\pm 0.009)$	$0.785 (\pm 0.005)$	$0.776 (\pm 0.002)$
$D \to K$	$0.777 (\pm 0.015)$	$0.79 (\pm 0.015)$	$0.79 (\pm 0.015)$	$0.79 (\pm 0.015)$	$0.791 (\pm 0.013$	$0.775 (\pm 0.014)$	$0.775 (\pm 0.014)$	$0.789 (\pm 0.013)$	$0.786 (\pm 0.013)$
$E \to B$	$0.688 (\pm 0.015)$	$0.709 (\pm 0.022$	$0.709 (\pm 0.022)$	$0.705 (\pm 0.024)$	$0.709 (\pm 0.021$	$0.704 (\pm {0.022}^{\circ}$	$0.695 (\pm 0.022)$	$0.712 (\pm 0.021)$	$0.707 (\pm 0.021$
$E \to D$	$0.714 (\pm 0.005)$	$0.734 (\pm 0.002)$	$0.734 (\pm 0.002)$	$0.737 (\pm 0.005$	$0.733 (\pm 0.008)$	$0.725 (\pm 0.004)$	$0.725 (\pm 0.004)$	$0.732 (\pm 0.006)$	$0.737 (\pm 0.011)$
$E \to K$	$0.858 (\pm 0.004)$	$0.874 (\pm 0.01)$	$0.874 (\pm 0.01)$	$0.874 (\pm 0.012)$	$0.876 (\pm 0.01)$	$0.86 (\pm 0.012)$	$0.86 (\pm 0.012)$	$0.875 (\pm 0.011)$	$0.865 (\pm 0.01)$
$K \to B$	$0.713 (\pm 0.002)$	$0.723 (\pm 0.006)$	$0.723 (\pm 0.006)$	$0.724 (\pm 0.009)$	$0.72 (\pm 0.004)$	$0.708 (\pm 0.014)$	$0.705 (\pm 0.011)$	$0.726 (\pm 0.006)$	$0.718 (\pm 0.006$
$K \to D$	$0.741 (\pm 0.01)$	$0.752 (\pm 0.012)$	$0.752 (\pm 0.012)$	$0.751 (\pm 0.013)$	$0.752 (\pm {0.017}^{\circ})$	$0.75 (\pm 0.019)$	$0.742 (\pm 0.008)$	$0.749 (\pm 0.012)$	$0.756 (\pm 0.014$
$K \to E$	$0.848 (\pm 0.008)$	$0.857 (\pm 0.007)$	$0.857 (\pm 0.007)$	$0.858 (\pm 0.008)$	$0.856 (\pm 0.008)$	$0.846 (\pm 0.012)$	$0.847 (\pm 0.007)$	$0.858 (\pm 0.008)$	$0.85 (\pm 0.008)$
平均值	$0.766 (\pm 0.009)$	$0.779 (\pm 0.01)$	$0.779 (\pm 0.01)$	$0.779 (\pm 0.011)$	$0.778 (\pm 0.01)$	$0.769 (\pm 0.011)$	$0.766 (\pm 0.011)$	$0.78 (\pm 0.01)$	$0.775 (\pm 0.011)$

Task	DIRT
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	1WV	DEV	IWA (ours)	TB
$B \to D$	$0.777 (\pm 0.01)$	$0.816 (\pm 0.008)$	$0.816 (\pm 0.008)$	$0.817 (\pm 0.008)$	$0.813 (\pm 0.007)$	$0.791 (\pm 0.012)$	$0.787 (\pm 0.032)$	$0.813 (\pm 0.004)$	$0.809 (\pm 0.008)$
$B \to E$	$0.756 (\pm 0.013$	$0.792 (\pm 0.006)$	$0.792 (\pm 0.006)$	$0.792 (\pm 0.01)$	$0.782 (\pm 0.008)$	$0.758 (\pm 0.011)$	$0.758 (\pm 0.011)$	$0.788 (\pm 0.008)$	$0.804 (\pm 0.005)$
$B \to K$	$0.77 (\pm 0.013)$	$0.815 (\pm 0.003$	$0.815 (\pm 0.003)$	$0.814 (\pm 0.005)$	$0.813 (\pm 0.006)$	$0.793 (\pm 0.009)$	$0.782 (\pm 0.025)$	$0.814 (\pm 0.003)$	$0.814 (\pm 0.003)$
$D \to B$	$0.773 (\pm 0.008)$	$0.812 (\pm 0.014)$	$0.812 (\pm 0.014)$	$0.809 (\pm 0.019)$	$0.814 (\pm 0.01)$	$0.805 (\pm 0.015)$	$0.808 (\pm 0.004)$	$0.81 (\pm 0.01)$	$0.811 (\pm 0.01)$
$D \to E$	$0.757 (\pm 0.02)$	$0.813 (\pm 0.009)$	$0.813 (\pm 0.009)$	$0.814 (\pm 0.012)$	$0.808 (\pm 0.012)$	$0.787 (\pm 0.028)$	$0.807 (\pm 0.024)$	$0.814 (\pm 0.009)$	$0.812 (\pm 0.009)$
$D \to K$	$0.775 (\pm 0.016)$	$0.836 (\pm 0.009)$	$0.836 (\pm 0.009)$	$0.835 (\pm 0.008)$	0.837(±0.00%	$0.789 (\pm 0.025)$	$0.787 (\pm 0.038)$	$0.832 (\pm 0.006)$	$0.828 (\pm 0.004)$
$E \to B$	$0.702 (\pm 0.018$	$0.672 (\pm 0.015)$	$0.672 (\pm 0.015)$	$0.674 (\pm {0.015}^{\circ})$	0.714(±0.016	0.702(±0.018	$0.675 (\pm 0.05)$	$0.678 (\pm 0.015)$	$0.708 (\pm 0.016)$
$E \to D$	$0.716 (\pm 0.008$	$0.689 (\pm 0.019)$	$0.689 (\pm 0.019)$	$0.69 (\pm 0.022)$	$0.731 (\pm 0.029)$	0.718(±0.01	$0.704 (\pm 0.027)$	$0.702 (\pm 0.021)$	$0.73 (\pm 0.029)$
$E \to K$	$0.855 (\pm 0.013)$	$0.888 (\pm 0.01)$	$0.888 (\pm 0.01)$	$0.885 (\pm 0.009)$	0.894(±0.005	$0.874 (\pm 0.008)$	$0.873 (\pm 0.009)$	$0.884 (\pm 0.009$	$0.889 (\pm 0.005)$
$K \to B$	$0.709 (\pm 0.012)$	$0.707 (\pm 0.007)$	$0.707 (\pm 0.007)$	$0.711 (\pm 0.003)$	$0.749 (\pm 0.005)$	$0.73 (\pm 0.02)$	$0.715 (\pm 0.005)$	$0.711 (\pm 0.008)$	0.723(±0.005’
$K \to D$	$0.736 (\pm 0.012)$	$0.721 (\pm 0.01)$	$0.721 (\pm 0.01)$	$0.727 (\pm 0.019)$	$0.779 (\pm 0.003)$	$0.745 (\pm 0.022)$	$0.731 (\pm 0.007)$	$0.727 (\pm 0.006)$	$0.756 (\pm 0.003)$
$K \to E$	$0.842 (\pm 0.009)$	$0.869 (\pm 0.01)$	$0.869 (\pm 0.01)$	$0.868 (\pm 0.01)$	$0.872 \hat{i} \pm 0.009$	$0.848 (\pm 0.012)$	$0.848 (\pm 0.008)$	$0.868 (\pm 0.01)$	$0.863 (\pm 0.009)$
Avg.	$0.764 (\pm 0.013)$	$0.786 (\pm 0.01)$	$0.786 (\pm 0.01)$	$0.786 (\pm 0.012)$	0.8(±0.01)	$0.778 (\pm 0.016)$	$0.773 (\pm 0.02)$	$0.787 (\pm 0.009)$	$0.795 (\pm 0.009)$

任务	DIRT
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	1WV	DEV	IWA（我们的）	TB
$B \to D$	$0.777 (\pm 0.01)$	$0.816 (\pm 0.008)$	$0.816 (\pm 0.008)$	$0.817 (\pm 0.008)$	$0.813 (\pm 0.007)$	$0.791 (\pm 0.012)$	$0.787 (\pm 0.032)$	$0.813 (\pm 0.004)$	$0.809 (\pm 0.008)$
$B \to E$	$0.756 (\pm 0.013$	$0.792 (\pm 0.006)$	$0.792 (\pm 0.006)$	$0.792 (\pm 0.01)$	$0.782 (\pm 0.008)$	$0.758 (\pm 0.011)$	$0.758 (\pm 0.011)$	$0.788 (\pm 0.008)$	$0.804 (\pm 0.005)$
$B \to K$	$0.77 (\pm 0.013)$	$0.815 (\pm 0.003$	$0.815 (\pm 0.003)$	$0.814 (\pm 0.005)$	$0.813 (\pm 0.006)$	$0.793 (\pm 0.009)$	$0.782 (\pm 0.025)$	$0.814 (\pm 0.003)$	$0.814 (\pm 0.003)$
$D \to B$	$0.773 (\pm 0.008)$	$0.812 (\pm 0.014)$	$0.812 (\pm 0.014)$	$0.809 (\pm 0.019)$	$0.814 (\pm 0.01)$	$0.805 (\pm 0.015)$	$0.808 (\pm 0.004)$	$0.81 (\pm 0.01)$	$0.811 (\pm 0.01)$
$D \to E$	$0.757 (\pm 0.02)$	$0.813 (\pm 0.009)$	$0.813 (\pm 0.009)$	$0.814 (\pm 0.012)$	$0.808 (\pm 0.012)$	$0.787 (\pm 0.028)$	$0.807 (\pm 0.024)$	$0.814 (\pm 0.009)$	$0.812 (\pm 0.009)$
$D \to K$	$0.775 (\pm 0.016)$	$0.836 (\pm 0.009)$	$0.836 (\pm 0.009)$	$0.835 (\pm 0.008)$	0.837（±0.00%	$0.789 (\pm 0.025)$	$0.787 (\pm 0.038)$	$0.832 (\pm 0.006)$	$0.828 (\pm 0.004)$
$E \to B$	$0.702 (\pm 0.018$	$0.672 (\pm 0.015)$	$0.672 (\pm 0.015)$	$0.674 (\pm {0.015}^{\circ})$	0.714（±0.016	0.702(±0.018	$0.675 (\pm 0.05)$	$0.678 (\pm 0.015)$	$0.708 (\pm 0.016)$
$E \to D$	$0.716 (\pm 0.008$	$0.689 (\pm 0.019)$	$0.689 (\pm 0.019)$	$0.69 (\pm 0.022)$	$0.731 (\pm 0.029)$	0.718(±0.01	$0.704 (\pm 0.027)$	$0.702 (\pm 0.021)$	$0.73 (\pm 0.029)$
$E \to K$	$0.855 (\pm 0.013)$	$0.888 (\pm 0.01)$	$0.888 (\pm 0.01)$	$0.885 (\pm 0.009)$	0.894(±0.005	$0.874 (\pm 0.008)$	$0.873 (\pm 0.009)$	$0.884 (\pm 0.009$	$0.889 (\pm 0.005)$
$K \to B$	$0.709 (\pm 0.012)$	$0.707 (\pm 0.007)$	$0.707 (\pm 0.007)$	$0.711 (\pm 0.003)$	$0.749 (\pm 0.005)$	$0.73 (\pm 0.02)$	$0.715 (\pm 0.005)$	$0.711 (\pm 0.008)$	0.723(±0.005’
$K \to D$	$0.736 (\pm 0.012)$	$0.721 (\pm 0.01)$	$0.721 (\pm 0.01)$	$0.727 (\pm 0.019)$	$0.779 (\pm 0.003)$	$0.745 (\pm 0.022)$	$0.731 (\pm 0.007)$	$0.727 (\pm 0.006)$	$0.756 (\pm 0.003)$
$K \to E$	$0.842 (\pm 0.009)$	$0.869 (\pm 0.01)$	$0.869 (\pm 0.01)$	$0.868 (\pm 0.01)$	$0.872 \hat{i} \pm 0.009$	$0.848 (\pm 0.012)$	$0.848 (\pm 0.008)$	$0.868 (\pm 0.01)$	$0.863 (\pm 0.009)$
平均值	$0.764 (\pm 0.013)$	$0.786 (\pm 0.01)$	$0.786 (\pm 0.01)$	$0.786 (\pm 0.012)$	0.8(±0.01)	$0.778 (\pm 0.016)$	$0.773 (\pm 0.02)$	$0.787 (\pm 0.009)$	$0.795 (\pm 0.009)$

Task	DDC
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.781 (\pm 0.016)$	$0.796 (\pm 0.008)$	$0.796 (\pm 0.008)$	$0.793 (\pm 0.013)$	$0.794 (\pm 0.002)$	$0.775 (\pm 0.017)$	$0.778 (\pm 0.02)$	$0.789 (\pm 0.009)$	$0.786 (\pm 0.008)$
$B \to E$	$0.752 (\pm 0.007)$	$0.762 (\pm 0.014)$	$0.762 (\pm 0.014)$	$0.762 (\pm 0.013)$	$0.766 (\pm 0.01)$	$0.752 (\pm 0.013)$	$0.749 (\pm 0.006)$	$0.763 (\pm 0.014)$	$0.756 (\pm 0.01)$
$B \to K$	$0.766 (\pm 0.015)$	$0.781 (\pm 0.015)$	$0.781 (\pm 0.015)$	$0.782 (\pm 0.017)$	$0.778 (\pm 0.02)$	$0.776 (\pm 0.016)$	$0.774 (\pm 0.008)$	$0.783 (\pm 0.014)$	$0.78 (\pm 0.014)$
$D \to B$	$0.781 (\pm 0.008)$	$0.798 (\pm 0.007$	$0.793 (\pm 0.007)$	$0.791 (\pm 0.006)$	$0.789 (\pm 0.012)$	$0.787 (\pm 0.009)$	$0.783 (\pm 0.005)$	$0.796 (\pm 0.006)$	$0.788 (\pm 0.006$
$D \to E$	$0.767 (\pm 0.015)$	$0.782 (\pm 0.004)$	$0.782 (\pm 0.004)$	$0.78 (\pm 0.004)$	$0.783 (\pm 0.008$	$0.778 (\pm 0.002)$	$0.776 (\pm 0.004)$	$0.784 (\pm 0.004$	$0.778 (\pm 0.004)$
$D \to K$	$0.782 (\pm 0.013)$	$0.792 (\pm 0.015)$	$0.792 (\pm 0.015)$	0.793(±0.015	$0.792 (\pm 0.018)$	$0.782 (\pm 0.014)$	$0.783 (\pm 0.014)$	$0.793 (\pm 0.017)$	$0.788 (\pm 0.017)$
$E \to B$	$0.693 (\pm 0.019)$	$0.703 (\pm 0.023)$	$0.703 (\pm 0.023)$	0.706(±0.021	$0.7 (\pm 0.023)$	$0.696 (\pm 0.011)$	$0.689 (\pm 0.021)$	$0.705 (\pm 0.025$	$0.704 (\pm 0.021)$
$E \to D$	$0.725 (\pm 0.006)$	0.736(±0.006	$0.736 (\pm 0.004)$	$0.731 (\pm 0.006)$	$0.735 (\pm 0.012)$	$0.711 (\pm 0.009)$	$0.736 (\pm 0.004)$	$0.736 (\pm 0.008$	$0.736 (\pm 0.007)$
$E \to K$	$0.857 (\pm 0.011)$	0.87%(±0.009	$0.874 (\pm 0.009)$	$0.872 (\pm 0.007)$	$0.874 (\pm 0.006)$	$0.857 (\pm 0.014)$	$0.853 (\pm 0.009)$	0.877(±0.011	$0.864 (\pm 0.011)$
$K \to B$	$0.713 (\pm 0.002)$	$0.728 (\pm 0.002$	$0.728 / \pm 0.002$	0.727(±0.005)	$0.721 (\pm 0.003)$	$0.709 (\pm 0.015)$	$0.708 \hat{i} \pm {0.007}^{\hat{i}}$	$0.726 \pm 0.003$	$0.719 (\pm 0.002)$
$K \to D$	$0.741 (\pm 0.018)$	$0.752 (\pm 0.011)$	$0.752 \hat{i} \pm 0.011 \hat{j}$	0.753(±0.01	$0.752 (\pm 0.016)$	$0.743 \hat{i} \pm 0.012$	$0.748 (\pm 0.01)$	$0.755 (\pm 0.013)$	$0.748 (\pm 0.012)$
$K \to E$	0.841(±0.001)	$0.855 (\pm 0.011)$	$0.855 (\pm 0.011)$	$0.856 (\pm 0.008)$	$0.858 \dot{6} \pm 0.008$	$0.84 (\pm 0.003)$	$0.84 (\pm 0.01)$	$0.856 (\pm 0.008$	$0.849 (\pm 0.008)$
Avg.	$0.766 (\pm 0.011)$	$0.78 (\pm 0.01)$	$0.78 (\pm 0.01)$	$0.779 (\pm 0.01)$	$0.778 (\pm 0.012)$	$0.767 (\pm 0.011)$	$0.768 (\pm 0.01)$	$0.78 (\pm 0.011)$	$0.775 (\pm 0.01)$

任务	DDC
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$B \to D$	$0.781 (\pm 0.016)$	$0.796 (\pm 0.008)$	$0.796 (\pm 0.008)$	$0.793 (\pm 0.013)$	$0.794 (\pm 0.002)$	$0.775 (\pm 0.017)$	$0.778 (\pm 0.02)$	$0.789 (\pm 0.009)$	$0.786 (\pm 0.008)$
$B \to E$	$0.752 (\pm 0.007)$	$0.762 (\pm 0.014)$	$0.762 (\pm 0.014)$	$0.762 (\pm 0.013)$	$0.766 (\pm 0.01)$	$0.752 (\pm 0.013)$	$0.749 (\pm 0.006)$	$0.763 (\pm 0.014)$	$0.756 (\pm 0.01)$
$B \to K$	$0.766 (\pm 0.015)$	$0.781 (\pm 0.015)$	$0.781 (\pm 0.015)$	$0.782 (\pm 0.017)$	$0.778 (\pm 0.02)$	$0.776 (\pm 0.016)$	$0.774 (\pm 0.008)$	$0.783 (\pm 0.014)$	$0.78 (\pm 0.014)$
$D \to B$	$0.781 (\pm 0.008)$	$0.798 (\pm 0.007$	$0.793 (\pm 0.007)$	$0.791 (\pm 0.006)$	$0.789 (\pm 0.012)$	$0.787 (\pm 0.009)$	$0.783 (\pm 0.005)$	$0.796 (\pm 0.006)$	$0.788 (\pm 0.006$
$D \to E$	$0.767 (\pm 0.015)$	$0.782 (\pm 0.004)$	$0.782 (\pm 0.004)$	$0.78 (\pm 0.004)$	$0.783 (\pm 0.008$	$0.778 (\pm 0.002)$	$0.776 (\pm 0.004)$	$0.784 (\pm 0.004$	$0.778 (\pm 0.004)$
$D \to K$	$0.782 (\pm 0.013)$	$0.792 (\pm 0.015)$	$0.792 (\pm 0.015)$	0.793(±0.015	$0.792 (\pm 0.018)$	$0.782 (\pm 0.014)$	$0.783 (\pm 0.014)$	$0.793 (\pm 0.017)$	$0.788 (\pm 0.017)$
$E \to B$	$0.693 (\pm 0.019)$	$0.703 (\pm 0.023)$	$0.703 (\pm 0.023)$	0.706(±0.021	$0.7 (\pm 0.023)$	$0.696 (\pm 0.011)$	$0.689 (\pm 0.021)$	$0.705 (\pm 0.025$	$0.704 (\pm 0.021)$
$E \to D$	$0.725 (\pm 0.006)$	0.736(±0.006	$0.736 (\pm 0.004)$	$0.731 (\pm 0.006)$	$0.735 (\pm 0.012)$	$0.711 (\pm 0.009)$	$0.736 (\pm 0.004)$	$0.736 (\pm 0.008$	$0.736 (\pm 0.007)$
$E \to K$	$0.857 (\pm 0.011)$	0.87%(±0.009	$0.874 (\pm 0.009)$	$0.872 (\pm 0.007)$	$0.874 (\pm 0.006)$	$0.857 (\pm 0.014)$	$0.853 (\pm 0.009)$	0.877(±0.011	$0.864 (\pm 0.011)$
$K \to B$	$0.713 (\pm 0.002)$	$0.728 (\pm 0.002$	$0.728 / \pm 0.002$	0.727(±0.005)	$0.721 (\pm 0.003)$	$0.709 (\pm 0.015)$	$0.708 \hat{i} \pm {0.007}^{\hat{i}}$	$0.726 \pm 0.003$	$0.719 (\pm 0.002)$
$K \to D$	$0.741 (\pm 0.018)$	$0.752 (\pm 0.011)$	$0.752 \hat{i} \pm 0.011 \hat{j}$	0.753(±0.01	$0.752 (\pm 0.016)$	$0.743 \hat{i} \pm 0.012$	$0.748 (\pm 0.01)$	$0.755 (\pm 0.013)$	$0.748 (\pm 0.012)$
$K \to E$	0.841(±0.001)	$0.855 (\pm 0.011)$	$0.855 (\pm 0.011)$	$0.856 (\pm 0.008)$	$0.858 \dot{6} \pm 0.008$	$0.84 (\pm 0.003)$	$0.84 (\pm 0.01)$	$0.856 (\pm 0.008$	$0.849 (\pm 0.008)$
平均值	$0.766 (\pm 0.011)$	$0.78 (\pm 0.01)$	$0.78 (\pm 0.01)$	$0.779 (\pm 0.01)$	$0.778 (\pm 0.012)$	$0.767 (\pm 0.011)$	$0.768 (\pm 0.01)$	$0.78 (\pm 0.011)$	$0.775 (\pm 0.01)$

Task	CMD
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.772 (\pm 0.009)$	$0.794 (\pm 0.008)$	$0.794 (\pm 0.008)$	$0.794 (\pm 0.004)$	$0.798 (\pm 0.003)$	$0.779 (\pm 0.018)$	$0.748 (\pm 0.036)$	0.8(±0.003)	$0.789 (\pm 0.003)$
$B \to E$	$0.745 (\pm 0.012)$	$0.785 (\pm 0.014)$	$0.785 (\pm 0.014)$	$0.779 (\pm 0.011)$	$0.779 (\pm 0.011)$	$0.746 (\pm 0.032)$	$0.651 (\pm 0.077)$	$0.782 (\pm 0.01)$	$0.78 (\pm 0.014)$
$B \to K$	$0.763 (\pm 0.015)$	$0.796 (\pm 0.01)$	$0.796 (\pm 0.01)$	$0.79 (\pm 0.007)$	$0.794 (\pm 0.013)$	$0.787 (\pm 0.011)$	$0.749 (\pm 0.028)$	$0.797 (\pm 0.007$	$0.793 (\pm 0.008$
$D \to B$	$0.788 (\pm 0.01)$	$0.801 (\pm 0.007)$	$0.801 (\pm 0.007)$	$0.798 (\pm 0.003)$	$0.804 (\pm 0.002$	$0.785 (\pm 0.011)$	$0.754 (\pm 0.022)$	$0.803 (\pm 0.008$	$0.794 (\pm 0.002)$
$D \to E$	$0.768 (\pm 0.003)$	$0.804 (\pm 0.006$	$0.804 (\pm 0.006)$	$0.798 (\pm 0.003)$	$0.801 (\pm 0.013)$	$0.777 (\pm 0.031)$	$0.744 (\pm 0.095)$	$0.802 (\pm 0.004$	$0.798 (\pm 0.007)$
$D \to K$	$0.777 (\pm 0.012)$	$0.803 (\pm 0.009)$	$0.803 (\pm 0.009)$	$0.797 (\pm 0.014)$	$0.804 (\pm 0.009)$	$0.773 (\pm {0.006}^{\circ}$	$0.673 (\pm 0.094)$	$0.801 (\pm 0.011$	$0.811 (\pm 0.007)$
$E \to B$	$0.699 (\pm 0.014)$	$0.72 (\pm 0.013)$	$0.72 (\pm 0.013)$	$0.717 (\pm 0.014)$	$0.724 (\pm 0.013$	$0.71 (\pm 0.016)$	$0.68 (\pm 0.039)$	$0.719 (\pm 0.013)$	$0.712 (\pm 0.013)$
$E \to D$	$0.722 (\pm 0.008)$	$0.752 (\pm 0.006)$	$0.752 (\pm 0.006)$	0.752(±0.004	$0.751 (\pm 0.005)$	$0.693 (\pm 0.041)$	$0.573 (\pm 0.01)$	$0.755 (\pm 0.003)$	$0.738 (\pm 0.003)$
$E \to K$	$0.86 (\pm 0.012)$	$0.874 (\pm 0.013)$	$0.874 (\pm 0.013)$	$0.872 (\pm 0.007)$	0.875(±0.008)	$0.822 (\pm 0.058)$	$0.807 (\pm 0.054)$	$0.876 (\pm 0.009)$	$0.871 (\pm 0.009)$
$K \to B$	$0.718 (\pm 0.005)$	$0.749 (\pm 0.007)$	$0.749 (\pm 0.007)$	$0.738 (\pm 0.003)$	$0.74 (\pm 0.003)$	$0.716 (\pm 0.022)$	$0.679 (\pm 0.056)$	$0.746 (\pm 0.004$	$0.74 (\pm 0.007)$
$K \to D$	$0.748 (\pm 0.003)$	$0.761 (\pm 0.016)$	$0.761 (\pm 0.016)$	$0.766 (\pm 0.022)$	0.77(±0.017)	$0.746 (\pm 0.011)$	$0.655 (\pm 0.119)$	$0.775 (\pm 0.016)$	$0.761 (\pm 0.016)$
$K \to E$	$0.842 (\pm 0.005)$	$0.864 (\pm 0.006)$	$0.864 (\pm 0.006)$	$0.862 (\pm 0.002)$	$0.869 (\pm 0.004)$	$0.85 (\pm 0.011)$	$0.806 (\pm 0.041)$	$0.867 (\pm 0.008)$	$0.858 (\pm 0.004)$
Avg.	$0.767 (\pm 0.009)$	$0.792 (\pm 0.01)$	$0.792 (\pm 0.01)$	$0.789 (\pm 0.008)$	$0.792 (\pm 0.008)$	$0.765 (\pm 0.022)$	$0.71 (\pm 0.056)$	$0.794 (\pm 0.008)$	$0.787 (\pm 0.008)$

任务	命令
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$B \to D$	$0.772 (\pm 0.009)$	$0.794 (\pm 0.008)$	$0.794 (\pm 0.008)$	$0.794 (\pm 0.004)$	$0.798 (\pm 0.003)$	$0.779 (\pm 0.018)$	$0.748 (\pm 0.036)$	0.8(±0.003)	$0.789 (\pm 0.003)$
$B \to E$	$0.745 (\pm 0.012)$	$0.785 (\pm 0.014)$	$0.785 (\pm 0.014)$	$0.779 (\pm 0.011)$	$0.779 (\pm 0.011)$	$0.746 (\pm 0.032)$	$0.651 (\pm 0.077)$	$0.782 (\pm 0.01)$	$0.78 (\pm 0.014)$
$B \to K$	$0.763 (\pm 0.015)$	$0.796 (\pm 0.01)$	$0.796 (\pm 0.01)$	$0.79 (\pm 0.007)$	$0.794 (\pm 0.013)$	$0.787 (\pm 0.011)$	$0.749 (\pm 0.028)$	$0.797 (\pm 0.007$	$0.793 (\pm 0.008$
$D \to B$	$0.788 (\pm 0.01)$	$0.801 (\pm 0.007)$	$0.801 (\pm 0.007)$	$0.798 (\pm 0.003)$	$0.804 (\pm 0.002$	$0.785 (\pm 0.011)$	$0.754 (\pm 0.022)$	$0.803 (\pm 0.008$	$0.794 (\pm 0.002)$
$D \to E$	$0.768 (\pm 0.003)$	$0.804 (\pm 0.006$	$0.804 (\pm 0.006)$	$0.798 (\pm 0.003)$	$0.801 (\pm 0.013)$	$0.777 (\pm 0.031)$	$0.744 (\pm 0.095)$	$0.802 (\pm 0.004$	$0.798 (\pm 0.007)$
$D \to K$	$0.777 (\pm 0.012)$	$0.803 (\pm 0.009)$	$0.803 (\pm 0.009)$	$0.797 (\pm 0.014)$	$0.804 (\pm 0.009)$	$0.773 (\pm {0.006}^{\circ}$	$0.673 (\pm 0.094)$	$0.801 (\pm 0.011$	$0.811 (\pm 0.007)$
$E \to B$	$0.699 (\pm 0.014)$	$0.72 (\pm 0.013)$	$0.72 (\pm 0.013)$	$0.717 (\pm 0.014)$	$0.724 (\pm 0.013$	$0.71 (\pm 0.016)$	$0.68 (\pm 0.039)$	$0.719 (\pm 0.013)$	$0.712 (\pm 0.013)$
$E \to D$	$0.722 (\pm 0.008)$	$0.752 (\pm 0.006)$	$0.752 (\pm 0.006)$	0.752(±0.004)	$0.751 (\pm 0.005)$	$0.693 (\pm 0.041)$	$0.573 (\pm 0.01)$	$0.755 (\pm 0.003)$	$0.738 (\pm 0.003)$
$E \to K$	$0.86 (\pm 0.012)$	$0.874 (\pm 0.013)$	$0.874 (\pm 0.013)$	$0.872 (\pm 0.007)$	0.875(±0.008)	$0.822 (\pm 0.058)$	$0.807 (\pm 0.054)$	$0.876 (\pm 0.009)$	$0.871 (\pm 0.009)$
$K \to B$	$0.718 (\pm 0.005)$	$0.749 (\pm 0.007)$	$0.749 (\pm 0.007)$	$0.738 (\pm 0.003)$	$0.74 (\pm 0.003)$	$0.716 (\pm 0.022)$	$0.679 (\pm 0.056)$	$0.746 (\pm 0.004$	$0.74 (\pm 0.007)$
$K \to D$	$0.748 (\pm 0.003)$	$0.761 (\pm 0.016)$	$0.761 (\pm 0.016)$	$0.766 (\pm 0.022)$	0.77(±0.017)	$0.746 (\pm 0.011)$	$0.655 (\pm 0.119)$	$0.775 (\pm 0.016)$	$0.761 (\pm 0.016)$
$K \to E$	$0.842 (\pm 0.005)$	$0.864 (\pm 0.006)$	$0.864 (\pm 0.006)$	$0.862 (\pm 0.002)$	$0.869 (\pm 0.004)$	$0.85 (\pm 0.011)$	$0.806 (\pm 0.041)$	$0.867 (\pm 0.008)$	$0.858 (\pm 0.004)$
平均值	$0.767 (\pm 0.009)$	$0.792 (\pm 0.01)$	$0.792 (\pm 0.01)$	$0.789 (\pm 0.008)$	$0.792 (\pm 0.008)$	$0.765 (\pm 0.022)$	$0.71 (\pm 0.056)$	$0.794 (\pm 0.008)$	$0.787 (\pm 0.008)$

Task	MMDA
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.775 (\pm 0.005)$	$0.796 (\pm 0.005)$	$0.796 (\pm 0.005)$	0.797(±0.009)	$0.794 (\pm 0.015)$	$0.783 (\pm 0.004)$	$0.786 (\pm 0.006)$	$0.797 (\pm 0.003)$	$0.792 (\pm 0.004)$
$B \to E$	$0.752 (\pm 0.006)$	$0.775 (\pm 0.01)$	$0.775 (\pm 0.01)$	$0.776 (\pm 0.008$	$0.771 (\pm 0.014)$	$0.766 (\pm 0.022)$	$0.756 (\pm 0.012)$	$0.779 (\pm 0.012)$	$0.781 (\pm 0.006)$
$B \to K$	$0.765 (\pm 0.009)$	$0.793 (\pm 0.016)$	$0.793 (\pm 0.016)$	$0.792 (\pm 0.017)$	$0.803 (\pm 0.019$	$0.763 (\pm 0.009)$	$0.767 (\pm 0.024)$	$0.795 (\pm 0.017)$	$0.802 (\pm 0.019)$
$D \to B$	$0.783 (\pm 0.007)$	$0.796 (\pm 0.007$	$0.796 (\pm 0.007)$	$0.795 (\pm 0.008)$	$0.796 (\pm 0.006)$	$0.789 (\pm 0.013)$	$0.789 (\pm 0.012)$	$0.798 (\pm 0.009)$	$0.79 (\pm 0.009)$
$D \to E$	$0.764 (\pm 0.003)$	$0.792 (\pm 0.005)$	$0.792 (\pm 0.005$	$0.792 (\pm 0.005)$	$0.788 (\pm 0.005)$	$0.775 (\pm 0.009$	$0.773 (\pm 0.009)$	$0.795 (\pm 0.006$	$0.799 (\pm 0.006)$
$D \to K$	$0.775 (\pm 0.012)$	$0.797 (\pm 0.014)$	$0.797 (\pm 0.014)$	$0.796 (\pm 0.013)$	$0.811 (\pm 0.008$	$0.791 (\pm {0.018}^{\circ}$	$0.79 (\pm 0.019)$	0.8(±0.012)	$0.8 (\pm 0.008)$
$E \to B$	$0.701 (\pm 0.015)$	$0.707 (\pm 0.017)$	$0.707 (\pm 0.017)$	$0.707 (\pm 0.021)$	0.713(±0.014)	$0.688 (\pm 0.022$	$0.699 (\pm 0.004)$	$0.709 (\pm 0.016$	$0.707 (\pm 0.014)$
$E \to D$	$0.738 (\pm 0.01)$	$0.741 (\pm 0.004)$	$0.741 (\pm 0.004)$	$0.744 (\pm 0.004$	$0.738 (\pm 0.001)$	$0.732 (\pm 0.007$	$0.703 (\pm 0.026)$	$0.749 (\pm 0.001$	$0.738 (\pm 0.001$
$E \to K$	$0.855 (\pm 0.015)$	$0.875 (\pm 0.007)$	$0.875 (\pm 0.007)$	$0.875 (\pm 0.007)$	$0.878 (\pm 0.008$	$0.852 (\pm 0.01)$	$0.856 (\pm 0.014)$	$0.878 (\pm 0.009$	$0.867 (\pm 0.009$
$K \to B$	$0.715 (\pm 0.008)$	$0.73 (\pm 0.003)$	$0.73 (\pm 0.003)$	$0.726 (\pm 0.002)$	$0.739 (\pm 0.006$	$0.72 (\pm 0.009)$	$0.716 (\pm 0.015)$	$0.732 (\pm 0.004$	$0.73 (\pm 0.006)$
$K \to D$	$0.736 (\pm 0.014)$	$0.761 (\pm 0.013)$	$0.761 (\pm 0.013)$	$0.755 (\pm 0.011)$	$0.764 (\pm 0.018)$	$0.734 (\pm 0.012)$	$0.728 (\pm 0.045)$	$0.758 (\pm 0.017)$	$0.75 (\pm 0.018)$
$K \to E$	$0.842 (\pm 0.008)$	$0.856 (\pm 0.009)$	$0.856 (\pm 0.009)$	$0.858 (\pm 0.009)$	$0.855 (\pm 0.012)$	$0.835 (\pm 0.01)$	$0.832 (\pm 0.014)$	$0.858 (\pm 0.005$	$0.847 (\pm 0.009)$
Avg.	$0.767 (\pm 0.009)$	$0.785 (\pm 0.009)$	$0.785 (\pm 0.009)$	$0.785 (\pm 0.01)$	$0.787 (\pm 0.01)$	$0.769 (\pm 0.012)$	$0.766 (\pm 0.017)$	$0.787 (\pm 0.009)$	$0.784 (\pm 0.009)$

任务	MMDA
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$B \to D$	$0.775 (\pm 0.005)$	$0.796 (\pm 0.005)$	$0.796 (\pm 0.005)$	0.797(±0.009)	$0.794 (\pm 0.015)$	$0.783 (\pm 0.004)$	$0.786 (\pm 0.006)$	$0.797 (\pm 0.003)$	$0.792 (\pm 0.004)$
$B \to E$	$0.752 (\pm 0.006)$	$0.775 (\pm 0.01)$	$0.775 (\pm 0.01)$	$0.776 (\pm 0.008$	$0.771 (\pm 0.014)$	$0.766 (\pm 0.022)$	$0.756 (\pm 0.012)$	$0.779 (\pm 0.012)$	$0.781 (\pm 0.006)$
$B \to K$	$0.765 (\pm 0.009)$	$0.793 (\pm 0.016)$	$0.793 (\pm 0.016)$	$0.792 (\pm 0.017)$	$0.803 (\pm 0.019$	$0.763 (\pm 0.009)$	$0.767 (\pm 0.024)$	$0.795 (\pm 0.017)$	$0.802 (\pm 0.019)$
$D \to B$	$0.783 (\pm 0.007)$	$0.796 (\pm 0.007$	$0.796 (\pm 0.007)$	$0.795 (\pm 0.008)$	$0.796 (\pm 0.006)$	$0.789 (\pm 0.013)$	$0.789 (\pm 0.012)$	$0.798 (\pm 0.009)$	$0.79 (\pm 0.009)$
$D \to E$	$0.764 (\pm 0.003)$	$0.792 (\pm 0.005)$	$0.792 (\pm 0.005$	$0.792 (\pm 0.005)$	$0.788 (\pm 0.005)$	$0.775 (\pm 0.009$	$0.773 (\pm 0.009)$	$0.795 (\pm 0.006$	$0.799 (\pm 0.006)$
$D \to K$	$0.775 (\pm 0.012)$	$0.797 (\pm 0.014)$	$0.797 (\pm 0.014)$	$0.796 (\pm 0.013)$	$0.811 (\pm 0.008$	$0.791 (\pm {0.018}^{\circ}$	$0.79 (\pm 0.019)$	0.8(±0.012)	$0.8 (\pm 0.008)$
$E \to B$	$0.701 (\pm 0.015)$	$0.707 (\pm 0.017)$	$0.707 (\pm 0.017)$	$0.707 (\pm 0.021)$	0.713(±0.014)	$0.688 (\pm 0.022$	$0.699 (\pm 0.004)$	$0.709 (\pm 0.016$	$0.707 (\pm 0.014)$
$E \to D$	$0.738 (\pm 0.01)$	$0.741 (\pm 0.004)$	$0.741 (\pm 0.004)$	$0.744 (\pm 0.004$	$0.738 (\pm 0.001)$	$0.732 (\pm 0.007$	$0.703 (\pm 0.026)$	$0.749 (\pm 0.001$	$0.738 (\pm 0.001$
$E \to K$	$0.855 (\pm 0.015)$	$0.875 (\pm 0.007)$	$0.875 (\pm 0.007)$	$0.875 (\pm 0.007)$	$0.878 (\pm 0.008$	$0.852 (\pm 0.01)$	$0.856 (\pm 0.014)$	$0.878 (\pm 0.009$	$0.867 (\pm 0.009$
$K \to B$	$0.715 (\pm 0.008)$	$0.73 (\pm 0.003)$	$0.73 (\pm 0.003)$	$0.726 (\pm 0.002)$	$0.739 (\pm 0.006$	$0.72 (\pm 0.009)$	$0.716 (\pm 0.015)$	$0.732 (\pm 0.004$	$0.73 (\pm 0.006)$
$K \to D$	$0.736 (\pm 0.014)$	$0.761 (\pm 0.013)$	$0.761 (\pm 0.013)$	$0.755 (\pm 0.011)$	$0.764 (\pm 0.018)$	$0.734 (\pm 0.012)$	$0.728 (\pm 0.045)$	$0.758 (\pm 0.017)$	$0.75 (\pm 0.018)$
$K \to E$	$0.842 (\pm 0.008)$	$0.856 (\pm 0.009)$	$0.856 (\pm 0.009)$	$0.858 (\pm 0.009)$	$0.855 (\pm 0.012)$	$0.835 (\pm 0.01)$	$0.832 (\pm 0.014)$	$0.858 (\pm 0.005$	$0.847 (\pm 0.009)$
平均值	$0.767 (\pm 0.009)$	$0.785 (\pm 0.009)$	$0.785 (\pm 0.009)$	$0.785 (\pm 0.01)$	$0.787 (\pm 0.01)$	$0.769 (\pm 0.012)$	$0.766 (\pm 0.017)$	$0.787 (\pm 0.009)$	$0.784 (\pm 0.009)$

Task	CoDATS
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$B \to D$	$0.783 (\pm 0.013)$	0.8(±0.002)	$0.8 (\pm 0.002)$	$0.8 (\pm 0.004)$	$0.803 (\pm 0.003)$	$0.788 (\pm 0.008)$	$0.792 (\pm 0.001)$	$0.805 (\pm 0.009)$	$0.801 (\pm 0.008)$
$B \to E$	$0.755 (\pm 0.005)$	$0.788 (\pm 0.004)$	$0.788 (\pm 0.004)$	$0.787 (\pm 0.005)$	$0.792 (\pm 0.008$	$0.771 (\pm 0.011)$	$0.771 (\pm 0.011)$	$0.788 (\pm 0.006$	$0.802 (\pm 0.009)$
$B \to K$	$0.771 (\pm 0.022)$	$0.808 (\pm 0.007$	$0.808 (\pm 0.007)$	$0.808 (\pm 0.008)$	$0.807 (\pm 0.008)$	$0.775 (\pm 0.022)$	$0.793 (\pm 0.023)$	$0.806 (\pm 0.009)$	$0.817 (\pm 0.007$
$D \to B$	$0.774 (\pm 0.001)$	$0.796 (\pm 0.005)$	$0.796 (\pm 0.005)$	$0.795 (\pm 0.005)$	$0.807 (\pm 0.007$	$0.791 (\pm 0.007$	$0.792 (\pm 0.007)$	$0.802 (\pm 0.004$	$0.8 (\pm 0.007)$
$D \to E$	$0.769 (\pm 0.003)$	$0.808 (\pm 0.009)$	$0.808 (\pm 0.009)$	$0.81 (\pm 0.009)$	$0.808 (\pm 0.008)$	$0.789 (\pm 0.02)$	$0.777 (\pm 0.008)$	$0.811 (\pm 0.009$	$0.817 (\pm 0.011)$
$D \to K$	$0.782 (\pm 0.021)$	$0.819 (\pm 0.011)$	$0.819 (\pm 0.011)$	$0.82 (\pm 0.01)$	$0.824 (\pm 0.01)$	$0.796 (\pm 0.021$	$0.789 (\pm 0.011)$	$0.82 (\pm 0.01)$	$0.828 (\pm 0.01)$
$E \to B$	$0.687 (\pm 0.013)$	$0.714 (\pm 0.019)$	$0.714 (\pm 0.019)$	$0.713 (\pm 0.015)$	0.725(±0.021	0.7(±0.027)	$0.688 (\pm 0.01)$	$0.717 (\pm 0.018)$	$0.723 (\pm 0.021)$
$E \to D$	$0.72 (\pm 0.013)$	$0.745 (\pm 0.023)$	$0.745 (\pm 0.023)$	$0.744 (\pm 0.026)$	0.756(±0.017)	$0.738 (\pm 0.025)$	$0.731 (\pm 0.027)$	$0.747 (\pm 0.025)$	$0.736 (\pm 0.017)$
$E \to K$	$0.859 (\pm 0.016)$	$0.883 (\pm 0.009$	$0.883 (\pm 0.009)$	$0.882 (\pm 0.008)$	$0.879 (\pm 0.007)$	$0.867 (\pm 0.01)$	$0.866 (\pm 0.01)$	$0.883 (\pm 0.012)$	$0.87 (\pm 0.012)$
$K \to B$	$0.712 (\pm 0.012)$	$0.732 (\pm 0.004)$	$0.732 (\pm 0.004)$	$0.732 (\pm 0.009)$	0.747(±0.012)	$0.726 (\pm 0.037)$	$0.694 (\pm 0.008)$	$0.737 (\pm 0.006)$	$0.751 (\pm 0.016)$
$K \to D$	$0.73 (\pm 0.022)$	$0.759 (\pm 0.006)$	$0.759 (\pm 0.006)$	$0.766 (\pm 0.009)$	$0.775 (\pm 0.016$	$0.752 (\pm 0.015)$	$0.738 (\pm 0.007)$	$0.766 (\pm 0.004)$	$0.764 (\pm 0.016)$
$K \to E$	$0.845 (\pm 0.0)$	$0.867 (\pm 0.01)$	$0.867 (\pm 0.01)$	$0.866 (\pm 0.011)$	0.868(±0.007	$0.853 (\pm 0.003)$	$0.851 (\pm 0.011)$	$0.867 (\pm 0.007)$	$0.856 (\pm 0.007)$
Avg.	$0.766 (\pm 0.012)$	$0.793 (\pm 0.009)$	$0.793 (\pm 0.009)$	$0.794 (\pm 0.01)$	$0.799 (\pm 0.01)$	$0.779 (\pm 0.017)$	$0.773 (\pm 0.011)$	$0.796 (\pm 0.01)$	$0.797 (\pm 0.012)$

任务	CoDATS
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$B \to D$	$0.783 (\pm 0.013)$	0.8(±0.002)	$0.8 (\pm 0.002)$	$0.8 (\pm 0.004)$	$0.803 (\pm 0.003)$	$0.788 (\pm 0.008)$	$0.792 (\pm 0.001)$	$0.805 (\pm 0.009)$	$0.801 (\pm 0.008)$
$B \to E$	$0.755 (\pm 0.005)$	$0.788 (\pm 0.004)$	$0.788 (\pm 0.004)$	$0.787 (\pm 0.005)$	$0.792 (\pm 0.008$	$0.771 (\pm 0.011)$	$0.771 (\pm 0.011)$	$0.788 (\pm 0.006$	$0.802 (\pm 0.009)$
$B \to K$	$0.771 (\pm 0.022)$	$0.808 (\pm 0.007$	$0.808 (\pm 0.007)$	$0.808 (\pm 0.008)$	$0.807 (\pm 0.008)$	$0.775 (\pm 0.022)$	$0.793 (\pm 0.023)$	$0.806 (\pm 0.009)$	$0.817 (\pm 0.007$
$D \to B$	$0.774 (\pm 0.001)$	$0.796 (\pm 0.005)$	$0.796 (\pm 0.005)$	$0.795 (\pm 0.005)$	$0.807 (\pm 0.007$	$0.791 (\pm 0.007$	$0.792 (\pm 0.007)$	$0.802 (\pm 0.004$	$0.8 (\pm 0.007)$
$D \to E$	$0.769 (\pm 0.003)$	$0.808 (\pm 0.009)$	$0.808 (\pm 0.009)$	$0.81 (\pm 0.009)$	$0.808 (\pm 0.008)$	$0.789 (\pm 0.02)$	$0.777 (\pm 0.008)$	$0.811 (\pm 0.009$	$0.817 (\pm 0.011)$
$D \to K$	$0.782 (\pm 0.021)$	$0.819 (\pm 0.011)$	$0.819 (\pm 0.011)$	$0.82 (\pm 0.01)$	$0.824 (\pm 0.01)$	$0.796 (\pm 0.021$	$0.789 (\pm 0.011)$	$0.82 (\pm 0.01)$	$0.828 (\pm 0.01)$
$E \to B$	$0.687 (\pm 0.013)$	$0.714 (\pm 0.019)$	$0.714 (\pm 0.019)$	$0.713 (\pm 0.015)$	0.725(±0.021	0.7(±0.027)	$0.688 (\pm 0.01)$	$0.717 (\pm 0.018)$	$0.723 (\pm 0.021)$
$E \to D$	$0.72 (\pm 0.013)$	$0.745 (\pm 0.023)$	$0.745 (\pm 0.023)$	$0.744 (\pm 0.026)$	0.756(±0.017)	$0.738 (\pm 0.025)$	$0.731 (\pm 0.027)$	$0.747 (\pm 0.025)$	$0.736 (\pm 0.017)$
$E \to K$	$0.859 (\pm 0.016)$	$0.883 (\pm 0.009$	$0.883 (\pm 0.009)$	$0.882 (\pm 0.008)$	$0.879 (\pm 0.007)$	$0.867 (\pm 0.01)$	$0.866 (\pm 0.01)$	$0.883 (\pm 0.012)$	$0.87 (\pm 0.012)$
$K \to B$	$0.712 (\pm 0.012)$	$0.732 (\pm 0.004)$	$0.732 (\pm 0.004)$	$0.732 (\pm 0.009)$	0.747(±0.012)	$0.726 (\pm 0.037)$	$0.694 (\pm 0.008)$	$0.737 (\pm 0.006)$	$0.751 (\pm 0.016)$
$K \to D$	$0.73 (\pm 0.022)$	$0.759 (\pm 0.006)$	$0.759 (\pm 0.006)$	$0.766 (\pm 0.009)$	$0.775 (\pm 0.016$	$0.752 (\pm 0.015)$	$0.738 (\pm 0.007)$	$0.766 (\pm 0.004)$	$0.764 (\pm 0.016)$
$K \to E$	$0.845 (\pm 0.0)$	$0.867 (\pm 0.01)$	$0.867 (\pm 0.01)$	$0.866 (\pm 0.011)$	0.868(±0.007	$0.853 (\pm 0.003)$	$0.851 (\pm 0.011)$	$0.867 (\pm 0.007)$	$0.856 (\pm 0.007)$
平均值	$0.766 (\pm 0.012)$	$0.793 (\pm 0.009)$	$0.793 (\pm 0.009)$	$0.794 (\pm 0.01)$	$0.799 (\pm 0.01)$	$0.779 (\pm 0.017)$	$0.773 (\pm 0.011)$	$0.796 (\pm 0.01)$	$0.797 (\pm 0.012)$

Task	Deep-Coral
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.778 (\pm 0.018)$	$0.799 (\pm 0.011)$	$0.799 (\pm 0.011)$	$0.799 (\pm 0.006)$	$0.794 (\pm 0.011)$	$0.792 (\pm 0.023)$	$0.789 (\pm 0.025)$	$0.801 (\pm 0.01)$	$0.794 (\pm 0.012)$
$B \to E$	$0.749 (\pm 0.013)$	$0.771 (\pm 0.01)$	$0.771 (\pm 0.01)$	$0.771 (\pm 0.008$	$0.764 (\pm 0.018)$	$0.759 (\pm 0.002)$	$0.752 (\pm 0.011)$	$0.773 (\pm 0.008$	$0.767 (\pm 0.008)$
$B \to K$	$0.769 (\pm 0.011)$	$0.785 (\pm 0.013)$	$0.785 (\pm 0.013)$	$0.784 (\pm 0.016)$	$0.783 (\pm 0.015)$	$0.762 (\pm {0.013}^{\circ}$	$0.77 (\pm 0.014)$	$0.785 (\pm 0.017)$	$0.779 (\pm 0.017)$
$D \to B$	$0.783 (\pm 0.009$	$0.796 (\pm 0.003)$	$0.796 (\pm 0.003)$	$0.796 (\pm 0.006)$	$0.792 (\pm 0.006)$	$0.78 (\pm 0.005)$	$0.773 (\pm 0.005)$	$0.797 (\pm 0.004$	$0.786 (\pm 0.004)$
$D \to E$	$0.76 (\pm 0.013)$	$0.785 (\pm 0.002)$	$0.785 (\pm 0.002)$	0.785(±0.002	$0.778 (\pm 0.006)$	$0.762 (\pm 0.006)$	$0.764 (\pm 0.008)$	0.784(±0.003	$0.785 (\pm 0.005)$
$D \to K$	$0.78 (\pm 0.016)$	$0.794 (\pm 0.011)$	$0.794 (\pm 0.011)$	$0.793 (\pm 0.012)$	0.795(±0.01%	$0.787 (\pm 0.007)$	$0.783 (\pm 0.019)$	0.795(±0.013	$0.789 (\pm 0.013)$
$E \to B$	$0.696 (\pm 0.009)$	$0.708 (\pm 0.022)$	0.708(±0.022)	0.709(±0.024	$0.708 (\pm 0.024)$	$0.688 (\pm 0.016)$	$0.698 (\pm 0.018)$	$0.71 (\pm 0.024)$	$0.708 (\pm 0.027)$
$E \to D$	$0.722 (\pm 0.009)$	$0.738 (\pm 0.005)$	$0.738 (\pm 0.005)$	$0.741 (\pm 0.005)$	0.741(±0.001	$0.73 (\pm 0.016)$	$0.72 (\pm 0.017)$	$0.74 (\pm 0.004)$	$0.739 (\pm 0.001)$
$E \to K$	$0.859 (\pm 0.01)$	$0.879 (\pm 0.005$	$0.879 (\pm 0.005)$	$0.878 (\pm 0.007)$	$0.877 (\pm 0.007)$	$0.857 (\pm 0.007)$	$0.858 (\pm 0.009)$	0.878(±0.007	$0.864 (\pm 0.005)$
$K \to B$	$0.72 (\pm 0.01)$	$0.733 (\pm 0.009)$	$0.733 (\pm 0.009)$	$0.736 \dot{1} \pm 0.009$	$0.733 (\pm 0.009)$	$0.719 (\pm 0.011)$	$0.722 (\pm 0.022)$	$0.739 (\pm 0.01)$	$0.74 (\pm 0.005)$
$K \to D$	$0.733 (\pm 0.011)$	$0.755 (\pm 0.012)$	$0.755 (\pm 0.012)$	$0.751 (\pm 0.009)$	0.759(±0.009	$0.75 (\pm 0.009)$	$0.752 (\pm {0.013}^{\circ})$	$0.755 (\pm 0.015)$	$0.751 (\pm 0.009)$
$K \to E$	$0.841 (\pm 0.01)$	$0.858 (\pm 0.009)$	$0.858 (\pm 0.009)$	0.858(±0.011	$0.857 (\pm 0.008)$	$0.844 (\pm 0.011)$	$0.849 (\pm 0.005)$	$0.857 (\pm 0.01)$	$0.849 (\pm 0.011)$
Avg.	$0.766 (\pm 0.012)$	$0.783 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.783 (\pm 0.01)$	$0.782 (\pm 0.011)$	$0.769 (\pm 0.011)$	$0.769 (\pm 0.014)$	$0.785 (\pm 0.01)$	$0.779 (\pm 0.01)$

任务	深度珊瑚
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$B \to D$	$0.778 (\pm 0.018)$	$0.799 (\pm 0.011)$	$0.799 (\pm 0.011)$	$0.799 (\pm 0.006)$	$0.794 (\pm 0.011)$	$0.792 (\pm 0.023)$	$0.789 (\pm 0.025)$	$0.801 (\pm 0.01)$	$0.794 (\pm 0.012)$
$B \to E$	$0.749 (\pm 0.013)$	$0.771 (\pm 0.01)$	$0.771 (\pm 0.01)$	$0.771 (\pm 0.008$	$0.764 (\pm 0.018)$	$0.759 (\pm 0.002)$	$0.752 (\pm 0.011)$	$0.773 (\pm 0.008$	$0.767 (\pm 0.008)$
$B \to K$	$0.769 (\pm 0.011)$	$0.785 (\pm 0.013)$	$0.785 (\pm 0.013)$	$0.784 (\pm 0.016)$	$0.783 (\pm 0.015)$	$0.762 (\pm {0.013}^{\circ}$	$0.77 (\pm 0.014)$	$0.785 (\pm 0.017)$	$0.779 (\pm 0.017)$
$D \to B$	$0.783 (\pm 0.009$	$0.796 (\pm 0.003)$	$0.796 (\pm 0.003)$	$0.796 (\pm 0.006)$	$0.792 (\pm 0.006)$	$0.78 (\pm 0.005)$	$0.773 (\pm 0.005)$	$0.797 (\pm 0.004$	$0.786 (\pm 0.004)$
$D \to E$	$0.76 (\pm 0.013)$	$0.785 (\pm 0.002)$	$0.785 (\pm 0.002)$	0.785(±0.002	$0.778 (\pm 0.006)$	$0.762 (\pm 0.006)$	$0.764 (\pm 0.008)$	0.784(±0.003	$0.785 (\pm 0.005)$
$D \to K$	$0.78 (\pm 0.016)$	$0.794 (\pm 0.011)$	$0.794 (\pm 0.011)$	$0.793 (\pm 0.012)$	0.795(±0.01%	$0.787 (\pm 0.007)$	$0.783 (\pm 0.019)$	0.795(±0.013	$0.789 (\pm 0.013)$
$E \to B$	$0.696 (\pm 0.009)$	$0.708 (\pm 0.022)$	0.708(±0.022)	0.709(±0.024	$0.708 (\pm 0.024)$	$0.688 (\pm 0.016)$	$0.698 (\pm 0.018)$	$0.71 (\pm 0.024)$	$0.708 (\pm 0.027)$
$E \to D$	$0.722 (\pm 0.009)$	$0.738 (\pm 0.005)$	$0.738 (\pm 0.005)$	$0.741 (\pm 0.005)$	0.741(±0.001	$0.73 (\pm 0.016)$	$0.72 (\pm 0.017)$	$0.74 (\pm 0.004)$	$0.739 (\pm 0.001)$
$E \to K$	$0.859 (\pm 0.01)$	$0.879 (\pm 0.005$	$0.879 (\pm 0.005)$	$0.878 (\pm 0.007)$	$0.877 (\pm 0.007)$	$0.857 (\pm 0.007)$	$0.858 (\pm 0.009)$	0.878(±0.007	$0.864 (\pm 0.005)$
$K \to B$	$0.72 (\pm 0.01)$	$0.733 (\pm 0.009)$	$0.733 (\pm 0.009)$	$0.736 \dot{1} \pm 0.009$	$0.733 (\pm 0.009)$	$0.719 (\pm 0.011)$	$0.722 (\pm 0.022)$	$0.739 (\pm 0.01)$	$0.74 (\pm 0.005)$
$K \to D$	$0.733 (\pm 0.011)$	$0.755 (\pm 0.012)$	$0.755 (\pm 0.012)$	$0.751 (\pm 0.009)$	0.759(±0.009	$0.75 (\pm 0.009)$	$0.752 (\pm {0.013}^{\circ})$	$0.755 (\pm 0.015)$	$0.751 (\pm 0.009)$
$K \to E$	$0.841 (\pm 0.01)$	$0.858 (\pm 0.009)$	$0.858 (\pm 0.009)$	0.858(±0.011	$0.857 (\pm 0.008)$	$0.844 (\pm 0.011)$	$0.849 (\pm 0.005)$	$0.857 (\pm 0.01)$	$0.849 (\pm 0.011)$
平均值	$0.766 (\pm 0.012)$	$0.783 (\pm 0.009)$	$0.783 (\pm 0.009)$	$0.783 (\pm 0.01)$	$0.782 (\pm 0.011)$	$0.769 (\pm 0.011)$	$0.769 (\pm 0.014)$	$0.785 (\pm 0.01)$	$0.779 (\pm 0.01)$

Task	CDAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.784 (\pm 0.022)$	$0.803 (\pm 0.007)$	$0.803 (\pm 0.007)$	$0.803 (\pm 0.004)$	$0.795 (\pm 0.009)$	$0.795 (\pm 0.009)$	$0.792 (\pm 0.006)$	$0.804 (\pm 0.009)$	$0.797 (\pm 0.006)$
$B \to E$	$0.758 (\pm 0.013)$	$0.776 (\pm 0.009)$	$0.776 (\pm 0.009$	$0.774 (\pm 0.009)$	$0.776 (\pm 0.012)$	$0.764 (\pm 0.012)$	$0.764 (\pm 0.012)$	$0.778 (\pm 0.009)$	$0.775 (\pm 0.009)$
$B \to K$	$0.769 (\pm 0.019)$	$0.796 (\pm 0.008)$	$0.796 (\pm 0.008$	$0.796 (\pm 0.008$	$0.792 (\pm 0.014)$	$0.78 (\pm 0.006)$	$0.791 (\pm 0.007)$	$0.796 (\pm 0.01)$	$0.796 (\pm 0.01)$
$D \to B$	$0.786 (\pm 0.01)$	$0.797 (\pm 0.007)$	$0.797 (\pm 0.007)$	$0.795 (\pm 0.004)$	$0.79 (\pm 0.004)$	$0.793 (\pm 0.01)$	$0.782 (\pm 0.01)$	$0.798 (\pm 0.007)$	$0.794 (\pm 0.008)$
$D \to E$	$0.761 (\pm 0.003)$	$0.8 (\pm 0.004)$	$0.8 (\pm 0.004)$	0.8(±0.006)	$0.8 (\pm 0.005)$	$0.775 (\pm 0.007)$	$0.77 (\pm 0.009)$	$0.799 (\pm 0.004$	$0.794 (\pm 0.006)$
$D \to K$	$0.778 (\pm 0.018)$	$0.797 (\pm 0.012)$	$0.797 (\pm 0.012)$	$0.797 (\pm 0.01)$	$0.804 (\pm 0.015)$	$0.792 (\pm 0.02)$	$0.793 (\pm 0.015)$	$0.801 (\pm 0.013$	$0.801 (\pm 0.015)$
$E \to B$	$0.692 (\pm 0.014)$	$0.707 (\pm 0.015)$	$0.707 (\pm 0.015)$	$0.707 (\pm 0.015)$	$0.713 (\pm 0.024)$	$0.695 (\pm 0.012$	$0.707 (\pm 0.021)$	$0.71 (\pm 0.018)$	$0.711 (\pm 0.024)$
$E \to D$	$0.72 (\pm 0.018)$	$0.741 (\pm 0.021)$	$0.741 (\pm 0.021)$	$0.738 (\pm 0.011)$	$0.741 (\pm 0.005$	$0.722 (\pm 0.017$	$0.723 (\pm 0.028)$	$0.741 (\pm 0.006$	$0.735 (\pm 0.009)$
$E \to K$	$0.86 (\pm 0.006)$	$0.879 (\pm 0.011)$	$0.879 (\pm 0.011)$	$0.878 (\pm 0.01)$	$0.874 (\pm 0.012)$	$0.861 (\pm 0.018$	$0.861 (\pm 0.018)$	$0.879 (\pm 0.011)$	$0.865 (\pm 0.011)$
$K \to B$	$0.706 (\pm 0.004)$	$0.727 (\pm 0.003)$	$0.727 (\pm 0.003)$	$0.728 (\pm 0.001)$	$0.732 (\pm 0.005$	$0.724 (\pm 0.011)$	$0.722 (\pm 0.004)$	$0.733 (\pm 0.004)$	$0.724 (\pm 0.004)$
$K \to D$	$0.748 (\pm 0.013)$	$0.765 (\pm 0.013)$	$0.765 (\pm 0.013)$	$0.764 (\pm 0.011)$	$0.766 (\pm 0.021$	$0.752 (\pm 0.021)$	$0.755 (\pm 0.005)$	$0.762 (\pm 0.01)$	$0.752 (\pm 0.021)$
$K \to E$	$0.845 (\pm 0.007)$	$0.86 (\pm 0.005)$	$0.86 (\pm 0.005)$	$0.861 (\pm 0.007)$	$0.863 (\pm 0.003$	$0.848 (\pm 0.005)$	$0.849 (\pm 0.007)$	$0.861 (\pm 0.006$	$0.85 (\pm 0.003)$
Avg.	$0.767 (\pm 0.012)$	$0.787 (\pm 0.01)$	$0.787 (\pm 0.01)$	$0.787 (\pm 0.008)$	$0.787 (\pm 0.011)$	$0.775 (\pm 0.012)$	$0.776 (\pm 0.012)$	$0.788 (\pm 0.009)$	$0.783 (\pm 0.01)$

任务	CDAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$B \to D$	$0.784 (\pm 0.022)$	$0.803 (\pm 0.007)$	$0.803 (\pm 0.007)$	$0.803 (\pm 0.004)$	$0.795 (\pm 0.009)$	$0.795 (\pm 0.009)$	$0.792 (\pm 0.006)$	$0.804 (\pm 0.009)$	$0.797 (\pm 0.006)$
$B \to E$	$0.758 (\pm 0.013)$	$0.776 (\pm 0.009)$	$0.776 (\pm 0.009$	$0.774 (\pm 0.009)$	$0.776 (\pm 0.012)$	$0.764 (\pm 0.012)$	$0.764 (\pm 0.012)$	$0.778 (\pm 0.009)$	$0.775 (\pm 0.009)$
$B \to K$	$0.769 (\pm 0.019)$	$0.796 (\pm 0.008)$	$0.796 (\pm 0.008$	$0.796 (\pm 0.008$	$0.792 (\pm 0.014)$	$0.78 (\pm 0.006)$	$0.791 (\pm 0.007)$	$0.796 (\pm 0.01)$	$0.796 (\pm 0.01)$
$D \to B$	$0.786 (\pm 0.01)$	$0.797 (\pm 0.007)$	$0.797 (\pm 0.007)$	$0.795 (\pm 0.004)$	$0.79 (\pm 0.004)$	$0.793 (\pm 0.01)$	$0.782 (\pm 0.01)$	$0.798 (\pm 0.007)$	$0.794 (\pm 0.008)$
$D \to E$	$0.761 (\pm 0.003)$	$0.8 (\pm 0.004)$	$0.8 (\pm 0.004)$	0.8(±0.006)	$0.8 (\pm 0.005)$	$0.775 (\pm 0.007)$	$0.77 (\pm 0.009)$	$0.799 (\pm 0.004$	$0.794 (\pm 0.006)$
$D \to K$	$0.778 (\pm 0.018)$	$0.797 (\pm 0.012)$	$0.797 (\pm 0.012)$	$0.797 (\pm 0.01)$	$0.804 (\pm 0.015)$	$0.792 (\pm 0.02)$	$0.793 (\pm 0.015)$	$0.801 (\pm 0.013$	$0.801 (\pm 0.015)$
$E \to B$	$0.692 (\pm 0.014)$	$0.707 (\pm 0.015)$	$0.707 (\pm 0.015)$	$0.707 (\pm 0.015)$	$0.713 (\pm 0.024)$	$0.695 (\pm 0.012$	$0.707 (\pm 0.021)$	$0.71 (\pm 0.018)$	$0.711 (\pm 0.024)$
$E \to D$	$0.72 (\pm 0.018)$	$0.741 (\pm 0.021)$	$0.741 (\pm 0.021)$	$0.738 (\pm 0.011)$	$0.741 (\pm 0.005$	$0.722 (\pm 0.017$	$0.723 (\pm 0.028)$	$0.741 (\pm 0.006$	$0.735 (\pm 0.009)$
$E \to K$	$0.86 (\pm 0.006)$	$0.879 (\pm 0.011)$	$0.879 (\pm 0.011)$	$0.878 (\pm 0.01)$	$0.874 (\pm 0.012)$	$0.861 (\pm 0.018$	$0.861 (\pm 0.018)$	$0.879 (\pm 0.011)$	$0.865 (\pm 0.011)$
$K \to B$	$0.706 (\pm 0.004)$	$0.727 (\pm 0.003)$	$0.727 (\pm 0.003)$	$0.728 (\pm 0.001)$	$0.732 (\pm 0.005$	$0.724 (\pm 0.011)$	$0.722 (\pm 0.004)$	$0.733 (\pm 0.004)$	$0.724 (\pm 0.004)$
$K \to D$	$0.748 (\pm 0.013)$	$0.765 (\pm 0.013)$	$0.765 (\pm 0.013)$	$0.764 (\pm 0.011)$	$0.766 (\pm 0.021$	$0.752 (\pm 0.021)$	$0.755 (\pm 0.005)$	$0.762 (\pm 0.01)$	$0.752 (\pm 0.021)$
$K \to E$	$0.845 (\pm 0.007)$	$0.86 (\pm 0.005)$	$0.86 (\pm 0.005)$	$0.861 (\pm 0.007)$	$0.863 (\pm 0.003$	$0.848 (\pm 0.005)$	$0.849 (\pm 0.007)$	$0.861 (\pm 0.006$	$0.85 (\pm 0.003)$
平均值	$0.767 (\pm 0.012)$	$0.787 (\pm 0.01)$	$0.787 (\pm 0.01)$	$0.787 (\pm 0.008)$	$0.787 (\pm 0.011)$	$0.775 (\pm 0.012)$	$0.776 (\pm 0.012)$	$0.788 (\pm 0.009)$	$0.783 (\pm 0.01)$

Task	DANN
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.783 (\pm 0.004)$	$0.794 (\pm 0.003)$	$0.794 (\pm 0.003)$	$0.794 (\pm 0.008)$	$0.803 (\pm 0.007)$	$0.782 (\pm 0.005)$	$0.782 (\pm 0.005)$	$0.804 (\pm 0.008)$	$0.813 (\pm 0.017)$
$B \to E$	$0.752 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.787 (\pm 0.009)$	$0.787 (\pm 0.013)$	$0.796 (\pm 0.006$	$0.783 (\pm 0.013)$	$0.782 (\pm 0.013)$	$0.789 (\pm 0.008)$	$0.8 (\pm 0.005)$
$B \to K$	$0.767 (\pm 0.022)$	$0.804 (\pm 0.007)$	$0.804 (\pm 0.007)$	$0.804 (\pm 0.008)$	$0.805 (\pm 0.004$	$0.796 (\pm 0.003$	$0.806 (\pm 0.015$	$0.804 (\pm 0.008)$	$0.82 (\pm 0.004)$
$D \to B$	$0.779 (\pm 0.002)$	$0.8 (\pm 0.007)$	$0.8 (\pm 0.007)$	$0.799 (\pm 0.006)$	$0.802 (\pm 0.003$	$0.796 (\pm {0.004}^{\circ}$	$0.778 (\pm 0.014)$	$0.804 (\pm 0.004$	$0.799 (\pm 0.004)$
$D \to E$	$0.767 (\pm 0.016)$	$0.805 (\pm 0.006)$	$0.805 (\pm 0.006)$	$0.807 (\pm 0.009$	$0.808 (\pm 0.012$	$0.775 (\pm 0.023)$	$0.78 (\pm 0.035)$	$0.81 (\pm 0.007)$	$0.815 (\pm 0.007)$
$D \to K$	$0.784 (\pm 0.013)$	$0.815 (\pm 0.008)$	$0.815 (\pm 0.008$	$0.817 (\pm 0.011)$	$0.829 (\pm 0.014$	$0.788 (\pm 0.007)$	$0.779 (\pm 0.012)$	$0.816 (\pm 0.011)$	$0.827 (\pm 0.014)$
$E \to B$	$0.701 (\pm 0.019)$	$0.712 (\pm 0.014)$	$0.712 (\pm 0.014)$	$0.712 (\pm 0.015)$	$0.724 (\pm 0.014$	$0.69 (\pm 0.03)$	$0.702 (\pm 0.022)$	0.712(±0.012	$0.721 (\pm 0.014)$
$E \to D$	$0.736 (\pm 0.005)$	$0.743 (\pm 0.014)$	$0.743 (\pm 0.014)$	$0.747 (\pm 0.012)$	$0.758 (\pm 0.012$	$0.735 (\pm 0.053)$	$0.738 (\pm 0.019)$	0.751(±0.008	$0.757 (\pm 0.012)$
$E \to K$	$0.854 (\pm 0.016)$	$0.877 (\pm 0.011)$	$0.877 (\pm 0.011)$	$0.878 (\pm 0.012)$	0.879(±0.014	$0.86 (\pm 0.009)$	$0.858 (\pm 0.007)$	$0.88 (\pm 0.013)$	$0.875 (\pm 0.012)$
$K \to B$	$0.711 (\pm 0.014)$	$0.741 (\pm 0.004)$	$0.741 (\pm 0.004)$	$0.74 (\pm 0.004)$	$0.759 (\pm 0.007)$	$0.718 (\pm 0.012)$	$0.729 (\pm 0.003)$	$0.75 (\pm 0.004)$	$0.747 \dot{(} \pm {0.007}^{\dot{}}$
$K \to D$	$0.738 (\pm 0.006)$	$0.768 (\pm 0.011)$	$0.768 (\pm 0.011)$	0.765(±0.011)	0.778(±0.013	$0.733 (\pm 0.013)$	$0.758 (\pm 0.019)$	$0.778 (\pm 0.012)$	$0.762 \dot{(} \pm {0.012}^{\dot{}}$
$K \to E$	$0.837 (\pm 0.013)$	$0.864 (\pm 0.008)$	$0.864 (\pm 0.008)$	$0.864 (\pm 0.007)$	$0.865 \hat{i} \pm 0.009$	$0.86 (\pm 0.013)$	$0.851 (\pm 0.008)$	$0.866 (\pm 0.011)$	$0.859 (\pm 0.011)$
Avg.	$0.767 (\pm 0.012)$	$0.792 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.793 (\pm 0.01)$	0.8(±0.01)	$0.776 (\pm 0.015)$	$0.778 (\pm 0.015)$	$0.797 (\pm 0.009)$	$0.8 (\pm 0.01)$

任务	DANN
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$B \to D$	$0.783 (\pm 0.004)$	$0.794 (\pm 0.003)$	$0.794 (\pm 0.003)$	$0.794 (\pm 0.008)$	$0.803 (\pm 0.007)$	$0.782 (\pm 0.005)$	$0.782 (\pm 0.005)$	$0.804 (\pm 0.008)$	$0.813 (\pm 0.017)$
$B \to E$	$0.752 (\pm 0.011)$	$0.787 (\pm 0.009)$	$0.787 (\pm 0.009)$	$0.787 (\pm 0.013)$	$0.796 (\pm 0.006$	$0.783 (\pm 0.013)$	$0.782 (\pm 0.013)$	$0.789 (\pm 0.008)$	$0.8 (\pm 0.005)$
$B \to K$	$0.767 (\pm 0.022)$	$0.804 (\pm 0.007)$	$0.804 (\pm 0.007)$	$0.804 (\pm 0.008)$	$0.805 (\pm 0.004$	$0.796 (\pm 0.003$	$0.806 (\pm 0.015$	$0.804 (\pm 0.008)$	$0.82 (\pm 0.004)$
$D \to B$	$0.779 (\pm 0.002)$	$0.8 (\pm 0.007)$	$0.8 (\pm 0.007)$	$0.799 (\pm 0.006)$	$0.802 (\pm 0.003$	$0.796 (\pm {0.004}^{\circ}$	$0.778 (\pm 0.014)$	$0.804 (\pm 0.004$	$0.799 (\pm 0.004)$
$D \to E$	$0.767 (\pm 0.016)$	$0.805 (\pm 0.006)$	$0.805 (\pm 0.006)$	$0.807 (\pm 0.009$	$0.808 (\pm 0.012$	$0.775 (\pm 0.023)$	$0.78 (\pm 0.035)$	$0.81 (\pm 0.007)$	$0.815 (\pm 0.007)$
$D \to K$	$0.784 (\pm 0.013)$	$0.815 (\pm 0.008)$	$0.815 (\pm 0.008$	$0.817 (\pm 0.011)$	$0.829 (\pm 0.014$	$0.788 (\pm 0.007)$	$0.779 (\pm 0.012)$	$0.816 (\pm 0.011)$	$0.827 (\pm 0.014)$
$E \to B$	$0.701 (\pm 0.019)$	$0.712 (\pm 0.014)$	$0.712 (\pm 0.014)$	$0.712 (\pm 0.015)$	$0.724 (\pm 0.014$	$0.69 (\pm 0.03)$	$0.702 (\pm 0.022)$	0.712(±0.012)	$0.721 (\pm 0.014)$
$E \to D$	$0.736 (\pm 0.005)$	$0.743 (\pm 0.014)$	$0.743 (\pm 0.014)$	$0.747 (\pm 0.012)$	$0.758 (\pm 0.012$	$0.735 (\pm 0.053)$	$0.738 (\pm 0.019)$	0.751(±0.008)	$0.757 (\pm 0.012)$
$E \to K$	$0.854 (\pm 0.016)$	$0.877 (\pm 0.011)$	$0.877 (\pm 0.011)$	$0.878 (\pm 0.012)$	0.879(±0.014)	$0.86 (\pm 0.009)$	$0.858 (\pm 0.007)$	$0.88 (\pm 0.013)$	$0.875 (\pm 0.012)$
$K \to B$	$0.711 (\pm 0.014)$	$0.741 (\pm 0.004)$	$0.741 (\pm 0.004)$	$0.74 (\pm 0.004)$	$0.759 (\pm 0.007)$	$0.718 (\pm 0.012)$	$0.729 (\pm 0.003)$	$0.75 (\pm 0.004)$	$0.747 \dot{(} \pm {0.007}^{\dot{}}$
$K \to D$	$0.738 (\pm 0.006)$	$0.768 (\pm 0.011)$	$0.768 (\pm 0.011)$	0.765(±0.011)	0.778(±0.013)	$0.733 (\pm 0.013)$	$0.758 (\pm 0.019)$	$0.778 (\pm 0.012)$	$0.762 \dot{(} \pm {0.012}^{\dot{}}$
$K \to E$	$0.837 (\pm 0.013)$	$0.864 (\pm 0.008)$	$0.864 (\pm 0.008)$	$0.864 (\pm 0.007)$	$0.865 \hat{i} \pm 0.009$	$0.86 (\pm 0.013)$	$0.851 (\pm 0.008)$	$0.866 (\pm 0.011)$	$0.859 (\pm 0.011)$
平均值	$0.767 (\pm 0.012)$	$0.792 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.793 (\pm 0.01)$	0.8(±0.01)	$0.776 (\pm 0.015)$	$0.778 (\pm 0.015)$	$0.797 (\pm 0.009)$	$0.8 (\pm 0.01)$

Task	DSAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$B \to D$	$0.779 (\pm 0.013)$	$0.801 (\pm 0.014)$	$0.801 (\pm 0.014)$	$0.803 (\pm 0.015)$	$0.807 (\pm 0.01)$	$0.79 (\pm 0.02)$	$0.689 (\pm 0.108)$	$0.805 (\pm 0.014$	$0.805 (\pm 0.01)$
$B \to E$	$0.752 (\pm 0.005)$	$0.788 (\pm 0.005)$	$0.788 (\pm 0.005)$	$0.783 (\pm 0.005)$	$0.798 (\pm 0.002)$	$0.775 (\pm 0.022)$	$0.764 (\pm 0.02)$	$0.787 (\pm 0.006$	$0.796 (\pm 0.002)$
$B \to K$	$0.768 (\pm 0.012)$	$0.799 (\pm 0.008)$	$0.799 (\pm 0.008)$	$0.799 (\pm 0.01)$	$0.815 (\pm 0.007)$	$0.786 (\pm 0.036$	$0.798 (\pm 0.031)$	0.801(±0.011	$0.816 (\pm 0.005$
$D \to B$	$0.782 (\pm 0.008)$	$0.795 (\pm 0.004)$	$0.795 (\pm 0.004)$	$0.795 (\pm 0.002)$	0.799\{±0.007	$0.797 (\pm 0.01)$	$0.729 (\pm 0.111)$	$0.801 (\pm 0.005$	$0.802 (\pm 0.007$
$D \to E$	$0.771 (\pm 0.002)$	$0.804 (\pm 0.008$	$0.804 (\pm 0.008)$	$0.804 (\pm 0.008$	$0.816 (\pm 0.012)$	$0.79 (\pm 0.02)$	$0.792 (\pm 0.02)$	$0.808 (\pm 0.003$	$0.813 (\pm 0.012$
$D \to K$	$0.786 (\pm 0.013)$	$0.804 (\pm 0.01)$	$0.804 (\pm 0.01)$	$0.802 (\pm 0.012)$	$0.821 (\pm 0.014)$	$0.789 (\pm 0.013)$	$0.801 (\pm 0.025$	$0.807 (\pm 0.012$	$0.83 (\pm 0.016)$
$E \to B$	$0.702 (\pm 0.021)$	$0.716 (\pm 0.017)$	$0.716 (\pm 0.017)$	$0.718 (\pm 0.021)$	$0.718 (\pm 0.018$	$0.71 (\pm 0.021)$	$0.707 (\pm 0.018$	$0.721 (\pm 0.02)$	$0.711 (\pm 0.02)$
$E \to D$	$0.725 (\pm 0.007)$	$0.743 (\pm 0.008)$	$0.743 (\pm 0.008)$	$0.741 (\pm 0.005$	$0.744 (\pm 0.008$	$0.723 (\pm 0.021$	$0.735 (\pm 0.012)$	$0.749 (\pm 0.005$	$0.732 (\pm 0.005$
$E \to K$	$0.865 (\pm 0.013)$	$0.883 (\pm 0.009$	$0.883 (\pm 0.009)$	$0.882 (\pm 0.008$	$0.886 (\pm 0.009)$	$0.874 (\pm 0.003)$	$0.876 (\pm 0.006$	$0.883 (\pm 0.01)$	$0.876 (\pm 0.009)$
$K \to B$	$0.716 (\pm 0.016)$	$0.741 (\pm 0.01)$	$0.741 (\pm 0.01)$	$0.736 (\pm 0.007)$	$0.746 (\pm 0.011)$	$0.712 (\pm 0.014)$	$0.707 (\pm 0.016)$	$0.742 (\pm 0.008$	$0.739 (\pm 0.011)$
$K \to D$	$0.739 (\pm 0.023)$	$0.773 (\pm 0.006)$	$0.773 (\pm 0.006)$	$0.768 (\pm 0.003)$	$0.777 (\pm 0.01)$	$0.746 (\pm 0.016)$	$0.726 (\pm 0.035)$	$0.776 (\pm 0.005$	$0.763 (\pm 0.012)$
$K \to E$	$0.84 (\pm 0.01)$	$0.86 (\pm 0.005)$	$0.86 (\pm 0.005)$	$0.858 (\pm 0.007)$	$0.868 (\pm 0.005)$	$0.854 (\pm 0.001)$	$0.835 (\pm 0.04)$	$0.865 (\pm 0.008$	$0.857 (\pm 0.005)$
Avg.	$0.769 (\pm 0.012)$	$0.792 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.791 (\pm 0.009)$	$0.8 (\pm 0.009)$	$0.779 (\pm 0.016)$	$0.763 (\pm 0.037)$	$0.795 (\pm 0.009)$	$0.795 (\pm 0.009)$

任务	DSAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$B \to D$	$0.779 (\pm 0.013)$	$0.801 (\pm 0.014)$	$0.801 (\pm 0.014)$	$0.803 (\pm 0.015)$	$0.807 (\pm 0.01)$	$0.79 (\pm 0.02)$	$0.689 (\pm 0.108)$	$0.805 (\pm 0.014$	$0.805 (\pm 0.01)$
$B \to E$	$0.752 (\pm 0.005)$	$0.788 (\pm 0.005)$	$0.788 (\pm 0.005)$	$0.783 (\pm 0.005)$	$0.798 (\pm 0.002)$	$0.775 (\pm 0.022)$	$0.764 (\pm 0.02)$	$0.787 (\pm 0.006$	$0.796 (\pm 0.002)$
$B \to K$	$0.768 (\pm 0.012)$	$0.799 (\pm 0.008)$	$0.799 (\pm 0.008)$	$0.799 (\pm 0.01)$	$0.815 (\pm 0.007)$	$0.786 (\pm 0.036$	$0.798 (\pm 0.031)$	0.801(±0.011	$0.816 (\pm 0.005$
$D \to B$	$0.782 (\pm 0.008)$	$0.795 (\pm 0.004)$	$0.795 (\pm 0.004)$	$0.795 (\pm 0.002)$	0.799{±0.007	$0.797 (\pm 0.01)$	$0.729 (\pm 0.111)$	$0.801 (\pm 0.005$	$0.802 (\pm 0.007$
$D \to E$	$0.771 (\pm 0.002)$	$0.804 (\pm 0.008$	$0.804 (\pm 0.008)$	$0.804 (\pm 0.008$	$0.816 (\pm 0.012)$	$0.79 (\pm 0.02)$	$0.792 (\pm 0.02)$	$0.808 (\pm 0.003$	$0.813 (\pm 0.012$
$D \to K$	$0.786 (\pm 0.013)$	$0.804 (\pm 0.01)$	$0.804 (\pm 0.01)$	$0.802 (\pm 0.012)$	$0.821 (\pm 0.014)$	$0.789 (\pm 0.013)$	$0.801 (\pm 0.025$	$0.807 (\pm 0.012$	$0.83 (\pm 0.016)$
$E \to B$	$0.702 (\pm 0.021)$	$0.716 (\pm 0.017)$	$0.716 (\pm 0.017)$	$0.718 (\pm 0.021)$	$0.718 (\pm 0.018$	$0.71 (\pm 0.021)$	$0.707 (\pm 0.018$	$0.721 (\pm 0.02)$	$0.711 (\pm 0.02)$
$E \to D$	$0.725 (\pm 0.007)$	$0.743 (\pm 0.008)$	$0.743 (\pm 0.008)$	$0.741 (\pm 0.005$	$0.744 (\pm 0.008$	$0.723 (\pm 0.021$	$0.735 (\pm 0.012)$	$0.749 (\pm 0.005$	$0.732 (\pm 0.005$
$E \to K$	$0.865 (\pm 0.013)$	$0.883 (\pm 0.009$	$0.883 (\pm 0.009)$	$0.882 (\pm 0.008$	$0.886 (\pm 0.009)$	$0.874 (\pm 0.003)$	$0.876 (\pm 0.006$	$0.883 (\pm 0.01)$	$0.876 (\pm 0.009)$
$K \to B$	$0.716 (\pm 0.016)$	$0.741 (\pm 0.01)$	$0.741 (\pm 0.01)$	$0.736 (\pm 0.007)$	$0.746 (\pm 0.011)$	$0.712 (\pm 0.014)$	$0.707 (\pm 0.016)$	$0.742 (\pm 0.008$	$0.739 (\pm 0.011)$
$K \to D$	$0.739 (\pm 0.023)$	$0.773 (\pm 0.006)$	$0.773 (\pm 0.006)$	$0.768 (\pm 0.003)$	$0.777 (\pm 0.01)$	$0.746 (\pm 0.016)$	$0.726 (\pm 0.035)$	$0.776 (\pm 0.005$	$0.763 (\pm 0.012)$
$K \to E$	$0.84 (\pm 0.01)$	$0.86 (\pm 0.005)$	$0.86 (\pm 0.005)$	$0.858 (\pm 0.007)$	$0.868 (\pm 0.005)$	$0.854 (\pm 0.001)$	$0.835 (\pm 0.04)$	$0.865 (\pm 0.008$	$0.857 (\pm 0.005)$
平均值	$0.769 (\pm 0.012)$	$0.792 (\pm 0.009)$	$0.792 (\pm 0.009)$	$0.791 (\pm 0.009)$	$0.8 (\pm 0.009)$	$0.779 (\pm 0.016)$	$0.763 (\pm 0.037)$	$0.795 (\pm 0.009)$	$0.795 (\pm 0.009)$

Method	MiniDomainNet
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
HoMM	$0.509 (\pm 0.018)$	$0.526 (\pm 0.009)$	$0.524 (\pm 0.010)$	$0.523 (\pm 0.010)$	$0.518 (\pm 0.015)$	$0.511 (\pm 0.018)$	$0.511 (\pm 0.018)$	$0.531 (\pm 0.007)$	$0.537 (\pm 0.020)$
AdvSKM	$0.509 (\pm 0.015)$	$0.516 (\pm 0.011)$	$0.514 (\pm 0.014)$	$0.514 (\pm 0.017)$	$0.514 (\pm 0.009)$	$0.515 (\pm 0.032)$	$0.512 (\pm 0.032)$	$0.516 (\pm 0.011$	$0.522 (\pm 0.017)$
DIRT	$0.499 (\pm 0.022)$	$0.517 (\pm 0.006)$	$0.515 (\pm 0.009)$	$0.508 (\pm 0.006)$	$0.507 (\pm 0.018)$	$0.493 (\pm 0.028)$	$0.498 (\pm 0.033)$	0.519(±0.006)	$0.525 (\pm 0.020)$
DDC	$0.510 (\pm 0.032)$	$0.514 (\pm 0.012)$	$0.512 (\pm 0.018)$	$0.511 (\pm 0.019$	$0.514 (\pm 0.015)$	$0.511 (\pm 0.028)$	$0.512 (\pm 0.043)$	$0.516 (\pm 0.015$	$0.521 (\pm 0.019)$
CMD	$0.509 (\pm 0.019)$	$0.528 (\pm 0.011)$	$0.531 (\pm 0.016$	$0.531 (\pm 0.016)$	$0.522 (\pm 0.013)$	$0.518 (\pm 0.025)$	$0.490 (\pm 0.037)$	$0.533 (\pm 0.009$	$0.533 (\pm 0.024)$
MMDA	$0.509 (\pm 0.021)$	0.524(±0.004	$0.522 (\pm 0.005)$	$0.523 (\pm 0.007)$	$0.519 (\pm 0.012)$	$0.517 (\pm 0.008)$	$0.524 (\pm 0.006)$	$0.527 (\pm 0.009$	$0.531 (\pm 0.029)$
CoDATS	$0.502 \pm 0.026$	$0.535 (\pm 0.026)$	$0.533 (\pm 0.027)$	0.536(±0.028	$0.525 (\pm 0.015)$	0.518(±0.031)	0.524(±0.051)	$0.536 (\pm 0.025$	$0.529 (\pm 0.042)$
Deep-Coral	$0.505 (\pm 0.022)$	$0.538 (\pm 0.009)$	$0.534 (\pm 0.013)$	$0.533 (\pm 0.012)$	$0.520 (\pm {0.010}^{\circ})$	$0.518 (\pm 0.022)$	$0.524 (\pm 0.034)$	$0.539 (\pm 0.012)$	$0.535 (\pm 0.024)$
CDAN	$0.514 (\pm 0.028)$	$0.526 (\pm 0.013$	$0.524 (\pm 0.011)$	$0.525 (\pm 0.008)$	$0.517 (\pm 0.005)$	$0.513 (\pm 0.011)$	$0.516 (\pm 0.014)$	$0.532 (\pm 0.010$	$0.542 (\pm 0.017)$
DANN	$0.496 (\pm 0.023)$	$0.530 (\pm 0.011)$	$0.535 (\pm 0.011)$	0.538(±0.005	$0.516 (\pm 0.011)$	$0.519 (\pm 0.015)$	$0.515 (\pm 0.013)$	$0.541 (\pm 0.006)$	$0.532 (\pm 0.014)$
DSAN	$0.509 (\pm 0.022)$	$0.537 (\pm 0.011)$	$0.534 (\pm 0.013)$	$0.543 (\pm θ, 011)$	$0.525 (\pm 0.013)$	$0.513 (\pm 0.018)$	$0.535 (\pm 0.023)$	$0.546 (\pm 0.007)$	$0.563 (\pm 0.013)$
Avg.	$0.507 (\pm 0.022)$	$0.526 (\pm 0.011)$	$0.525 (\pm 0.014)$	$0.526 (\pm 0.013)$	$0.518 (\pm 0.012)$	$0.513 (\pm 0.022)$	$0.515 (\pm 0.028)$	$0.531 (\pm 0.011)$	$0.534 (\pm 0.022)$

方法	MiniDomainNet
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
HoMM	$0.509 (\pm 0.018)$	$0.526 (\pm 0.009)$	$0.524 (\pm 0.010)$	$0.523 (\pm 0.010)$	$0.518 (\pm 0.015)$	$0.511 (\pm 0.018)$	$0.511 (\pm 0.018)$	$0.531 (\pm 0.007)$	$0.537 (\pm 0.020)$
AdvSKM	$0.509 (\pm 0.015)$	$0.516 (\pm 0.011)$	$0.514 (\pm 0.014)$	$0.514 (\pm 0.017)$	$0.514 (\pm 0.009)$	$0.515 (\pm 0.032)$	$0.512 (\pm 0.032)$	$0.516 (\pm 0.011$	$0.522 (\pm 0.017)$
DIRT	$0.499 (\pm 0.022)$	$0.517 (\pm 0.006)$	$0.515 (\pm 0.009)$	$0.508 (\pm 0.006)$	$0.507 (\pm 0.018)$	$0.493 (\pm 0.028)$	$0.498 (\pm 0.033)$	0.519(±0.006)	$0.525 (\pm 0.020)$
DDC	$0.510 (\pm 0.032)$	$0.514 (\pm 0.012)$	$0.512 (\pm 0.018)$	$0.511 (\pm 0.019$	$0.514 (\pm 0.015)$	$0.511 (\pm 0.028)$	$0.512 (\pm 0.043)$	$0.516 (\pm 0.015$	$0.521 (\pm 0.019)$
CMD	$0.509 (\pm 0.019)$	$0.528 (\pm 0.011)$	$0.531 (\pm 0.016$	$0.531 (\pm 0.016)$	$0.522 (\pm 0.013)$	$0.518 (\pm 0.025)$	$0.490 (\pm 0.037)$	$0.533 (\pm 0.009$	$0.533 (\pm 0.024)$
MMDA	$0.509 (\pm 0.021)$	0.524(±0.004	$0.522 (\pm 0.005)$	$0.523 (\pm 0.007)$	$0.519 (\pm 0.012)$	$0.517 (\pm 0.008)$	$0.524 (\pm 0.006)$	$0.527 (\pm 0.009$	$0.531 (\pm 0.029)$
CoDATS	$0.502 \pm 0.026$	$0.535 (\pm 0.026)$	$0.533 (\pm 0.027)$	0.536(±0.028	$0.525 (\pm 0.015)$	0.518(±0.031)	0.524(±0.051)	$0.536 (\pm 0.025$	$0.529 (\pm 0.042)$
Deep-Coral	$0.505 (\pm 0.022)$	$0.538 (\pm 0.009)$	$0.534 (\pm 0.013)$	$0.533 (\pm 0.012)$	$0.520 (\pm {0.010}^{\circ})$	$0.518 (\pm 0.022)$	$0.524 (\pm 0.034)$	$0.539 (\pm 0.012)$	$0.535 (\pm 0.024)$
CDAN	$0.514 (\pm 0.028)$	$0.526 (\pm 0.013$	$0.524 (\pm 0.011)$	$0.525 (\pm 0.008)$	$0.517 (\pm 0.005)$	$0.513 (\pm 0.011)$	$0.516 (\pm 0.014)$	$0.532 (\pm 0.010$	$0.542 (\pm 0.017)$
DANN	$0.496 (\pm 0.023)$	$0.530 (\pm 0.011)$	$0.535 (\pm 0.011)$	0.538(±0.005	$0.516 (\pm 0.011)$	$0.519 (\pm 0.015)$	$0.515 (\pm 0.013)$	$0.541 (\pm 0.006)$	$0.532 (\pm 0.014)$
DSAN	$0.509 (\pm 0.022)$	$0.537 (\pm 0.011)$	$0.534 (\pm 0.013)$	$0.543 (\pm θ, 011)$	$0.525 (\pm 0.013)$	$0.513 (\pm 0.018)$	$0.535 (\pm 0.023)$	$0.546 (\pm 0.007)$	$0.563 (\pm 0.013)$
平均	$0.507 (\pm 0.022)$	$0.526 (\pm 0.011)$	$0.525 (\pm 0.014)$	$0.526 (\pm 0.013)$	$0.518 (\pm 0.012)$	$0.513 (\pm 0.022)$	$0.515 (\pm 0.028)$	$0.531 (\pm 0.011)$	$0.534 (\pm 0.022)$

Task				Heuristic	HoMM		Theoretical error guarantees
Task	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$R \to C$	$0.552 (\pm 0.041)$	$0.587 (\pm 0.017)$	$0.587 (\pm 0.017)$	$0.587 (\pm 0.005)$	$0.557 (\pm 0.008)$	$0.538 (\pm 0.029)$	$0.56 (\pm 0.017)$	$0.596 (\pm 0.009)$	$0.607 (\pm 0.022)$
$R \to I$	$0.373 (\pm 0.014)$	$0.376 (\pm 0.012)$	$0.376 (\pm 0.012)$	$0.373 (\pm 0.005)$	$0.386 (\pm 0.007$	$0.37 (\pm 0.01)$	$0.37 (\pm 0.01)$	$0.392 (\pm 0.007$	$0.413 (\pm 0.041)$
$R \to P$	$0.709 (\pm 0.026)$	$0.722 (\pm 0.01)$	$0.722 (\pm 0.01)$	$0.718 (\pm 0.006)$	$0.709 (\pm 0.011)$	$0.704 (\pm 0.013)$	$0.705 (\pm 0.014)$	$0.73 (\pm 0.003)$	$0.721 (\pm 0.003)$
$R \to O$	$0.332 (\pm 0.009)$	$0.35 (\pm 0.006)$	$0.35 (\pm 0.006)$	$0.353 (\pm 0.008)$	$0.356 (\pm 0.02)$	$0.367 (\pm 0.019)$	$0.342 (\pm 0.018)$	$0.353 (\pm 0.003)$	$0.37 (\pm 0.013)$
$R \to S$	$0.579 (\pm 0.022)$	$0.587 (\pm 0.012)$	$0.587 (\pm 0.012)$	$0.586 (\pm 0.003)$	$0.582 (\pm 0.01)$	$0.577 (\pm 0.004)$	$0.577 (\pm 0.004)$	$0.586 (\pm 0.008$	$0.609 (\pm 0.011)$
Avg.	$0.509 (\pm 0.023)$	$0.524 (\pm 0.011)$	$0.524 (\pm 0.011)$	$0.523 (\pm 0.005)$	$0.518 (\pm 0.011)$	$0.511 (\pm 0.015)$	$0.511 (\pm 0.013)$	$0.531 (\pm 0.006)$	$0.544 (\pm 0.018)$

任务				启发式	HoMM		理论误差保证
任务	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$R \to C$	$0.552 (\pm 0.041)$	$0.587 (\pm 0.017)$	$0.587 (\pm 0.017)$	$0.587 (\pm 0.005)$	$0.557 (\pm 0.008)$	$0.538 (\pm 0.029)$	$0.56 (\pm 0.017)$	$0.596 (\pm 0.009)$	$0.607 (\pm 0.022)$
$R \to I$	$0.373 (\pm 0.014)$	$0.376 (\pm 0.012)$	$0.376 (\pm 0.012)$	$0.373 (\pm 0.005)$	$0.386 (\pm 0.007$	$0.37 (\pm 0.01)$	$0.37 (\pm 0.01)$	$0.392 (\pm 0.007$	$0.413 (\pm 0.041)$
$R \to P$	$0.709 (\pm 0.026)$	$0.722 (\pm 0.01)$	$0.722 (\pm 0.01)$	$0.718 (\pm 0.006)$	$0.709 (\pm 0.011)$	$0.704 (\pm 0.013)$	$0.705 (\pm 0.014)$	$0.73 (\pm 0.003)$	$0.721 (\pm 0.003)$
$R \to O$	$0.332 (\pm 0.009)$	$0.35 (\pm 0.006)$	$0.35 (\pm 0.006)$	$0.353 (\pm 0.008)$	$0.356 (\pm 0.02)$	$0.367 (\pm 0.019)$	$0.342 (\pm 0.018)$	$0.353 (\pm 0.003)$	$0.37 (\pm 0.013)$
$R \to S$	$0.579 (\pm 0.022)$	$0.587 (\pm 0.012)$	$0.587 (\pm 0.012)$	$0.586 (\pm 0.003)$	$0.582 (\pm 0.01)$	$0.577 (\pm 0.004)$	$0.577 (\pm 0.004)$	$0.586 (\pm 0.008$	$0.609 (\pm 0.011)$
平均值	$0.509 (\pm 0.023)$	$0.524 (\pm 0.011)$	$0.524 (\pm 0.011)$	$0.523 (\pm 0.005)$	$0.518 (\pm 0.011)$	$0.511 (\pm 0.015)$	$0.511 (\pm 0.013)$	$0.531 (\pm 0.006)$	$0.544 (\pm 0.018)$

Task	AdvSKM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$R \to C$	$0.544 (\pm 0.017)$	$0.56 (\pm 0.025)$	$0.56 (\pm 0.025)$	$0.566 (\pm 0.03)$	$0.544 (\pm 0.024)$	$0.549 (\pm 0.022)$	$0.549 (\pm 0.022)$	$0.568 (\pm 0.009)$	$0.582 (\pm 0.046)$
$R \to I$	$0.38 (\pm 0.018)$	$0.373 (\pm 0.016)$	$0.373 (\pm 0.016)$	$0.371 (\pm 0.009)$	$0.394 (\pm 0.037)$	$0.394 (\pm 0.021)$	0.394(±0.021	$0.377 (\pm 0.013)$	$0.395 (\pm 0.02)$
$R \to P$	$0.723 (\pm 0.009)$	$0.721 (\pm 0.004$	$0.721 (\pm 0.004)$	$0.719 (\pm 0.003)$	$0.712 (\pm 0.006)$	$0.706 (\pm 0.041)$	$0.695 (\pm 0.034)$	$0.721 (\pm 0.003$	$0.723 (\pm 0.009)$
$R \to O$	$0.322 (\pm 0.027)$	$0.332 (\pm 0.006)$	$0.332 (\pm 0.006)$	$0.328 (\pm 0.005)$	0.335(±0.002	$0.335 / \pm 0.002$	$0.332 (\pm 0.006)$	$0.333 (\pm 0.005)$	$0.335 (\pm 0.004)$
$R \to S$	$0.579 (\pm 0.017)$	$0.586 (\pm 0.001)$	$0.586 (\pm 0.001)$	$0.584 (\pm 0.003)$	$0.586 (\pm 0.005)$	0.589(±0.007)	$0.589 (\pm 0.007)$	$0.582 (\pm 0.004)$	$0.592 (\pm 0.009)$
Avg.	$0.509 (\pm 0.018)$	$0.514 (\pm 0.01)$	$0.514 (\pm 0.01)$	$0.514 (\pm 0.01)$	$0.514 (\pm 0.015)$	$0.515 (\pm 0.018)$	$0.512 (\pm 0.018)$	$0.516 (\pm 0.007)$	$0.525 (\pm 0.018)$

任务	AdvSKM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$R \to C$	$0.544 (\pm 0.017)$	$0.56 (\pm 0.025)$	$0.56 (\pm 0.025)$	$0.566 (\pm 0.03)$	$0.544 (\pm 0.024)$	$0.549 (\pm 0.022)$	$0.549 (\pm 0.022)$	$0.568 (\pm 0.009)$	$0.582 (\pm 0.046)$
$R \to I$	$0.38 (\pm 0.018)$	$0.373 (\pm 0.016)$	$0.373 (\pm 0.016)$	$0.371 (\pm 0.009)$	$0.394 (\pm 0.037)$	$0.394 (\pm 0.021)$	0.394(±0.021)	$0.377 (\pm 0.013)$	$0.395 (\pm 0.02)$
$R \to P$	$0.723 (\pm 0.009)$	$0.721 (\pm 0.004$	$0.721 (\pm 0.004)$	$0.719 (\pm 0.003)$	$0.712 (\pm 0.006)$	$0.706 (\pm 0.041)$	$0.695 (\pm 0.034)$	$0.721 (\pm 0.003$	$0.723 (\pm 0.009)$
$R \to O$	$0.322 (\pm 0.027)$	$0.332 (\pm 0.006)$	$0.332 (\pm 0.006)$	$0.328 (\pm 0.005)$	0.335(±0.002)	$0.335 / \pm 0.002$	$0.332 (\pm 0.006)$	$0.333 (\pm 0.005)$	$0.335 (\pm 0.004)$
$R \to S$	$0.579 (\pm 0.017)$	$0.586 (\pm 0.001)$	$0.586 (\pm 0.001)$	$0.584 (\pm 0.003)$	$0.586 (\pm 0.005)$	0.589(±0.007)	$0.589 (\pm 0.007)$	$0.582 (\pm 0.004)$	$0.592 (\pm 0.009)$
平均值	$0.509 (\pm 0.018)$	$0.514 (\pm 0.01)$	$0.514 (\pm 0.01)$	$0.514 (\pm 0.01)$	$0.514 (\pm 0.015)$	$0.515 (\pm 0.018)$	$0.512 (\pm 0.018)$	$0.516 (\pm 0.007)$	$0.525 (\pm 0.018)$

Task	DIRT
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$R \to C$	$0.53 (\pm 0.013)$	$0.577 (\pm 0.033)$	$0.577 (\pm 0.033)$	$0.546 (\pm 0.031)$	$0.571 (\pm 0.017)$	$0.541 (\pm 0.025)$	$0.544 (\pm 0.025)$	$0.577 (\pm 0.025)$	$0.587 (\pm 0.037)$
$R \to I$	$0.373 (\pm 0.038)$	$0.365 (\pm 0.041)$	$0.365 (\pm 0.041)$	$0.374 (\pm 0.037)$	$0.37 (\pm 0.007)$	$0.36 (\pm 0.011)$	$0.385 (\pm 0.062)$	$0.38 (\pm 0.031)$	$0.431 (\pm 0.045)$
$R \to P$	$0.704 (\pm 0.031)$	0.708(±0.026)	$0.708 (\pm 0.026)$	$0.706 (\pm 0.02)$	$0.689 (\pm 0.015)$	$0.663 (\pm 0.03)$	$0.67 (\pm 0.042)$	$0.713 (\pm 0.03)$	$0.715 (\pm 0.028)$
$R \to O$	$0.34 (\pm 0.02)$	0.345(±0.031)	$0.345 (\pm 0.031)$	$0.34 (\pm 0.046)$	$0.336 (\pm 0.024)$	$0.326 (\pm 0.086)$	$0.365 (\pm 0.038$	$0.342 (\pm 0.022)$	$0.382 (\pm {0.027}^{\circ})$
$R \to S$	$0.55 (\pm 0.028)$	0.578(±0.006)	$0.578 (\pm 0.006)$	$0.571 (\pm 0.006)$	$0.571 (\pm 0.012)$	$0.572 (\pm 0.005)$	$0.525 (\pm 0.088)$	$0.583 (\pm 0.016$	$0.588 (\pm 0.033)$
Avg.	$0.499 (\pm 0.026)$	$0.515 (\pm 0.027)$	$0.515 (\pm 0.027)$	$0.508 (\pm 0.028)$	$0.507 (\pm 0.015)$	$0.493 (\pm 0.031)$	$0.498 (\pm 0.051)$	$0.519 (\pm 0.025)$	$0.541 (\pm 0.034)$

任务	DIRT
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$R \to C$	$0.53 (\pm 0.013)$	$0.577 (\pm 0.033)$	$0.577 (\pm 0.033)$	$0.546 (\pm 0.031)$	$0.571 (\pm 0.017)$	$0.541 (\pm 0.025)$	$0.544 (\pm 0.025)$	$0.577 (\pm 0.025)$	$0.587 (\pm 0.037)$
$R \to I$	$0.373 (\pm 0.038)$	$0.365 (\pm 0.041)$	$0.365 (\pm 0.041)$	$0.374 (\pm 0.037)$	$0.37 (\pm 0.007)$	$0.36 (\pm 0.011)$	$0.385 (\pm 0.062)$	$0.38 (\pm 0.031)$	$0.431 (\pm 0.045)$
$R \to P$	$0.704 (\pm 0.031)$	0.708（±0.026）	$0.708 (\pm 0.026)$	$0.706 (\pm 0.02)$	$0.689 (\pm 0.015)$	$0.663 (\pm 0.03)$	$0.67 (\pm 0.042)$	$0.713 (\pm 0.03)$	$0.715 (\pm 0.028)$
$R \to O$	$0.34 (\pm 0.02)$	0.345（±0.031）	$0.345 (\pm 0.031)$	$0.34 (\pm 0.046)$	$0.336 (\pm 0.024)$	$0.326 (\pm 0.086)$	$0.365 (\pm 0.038$	$0.342 (\pm 0.022)$	$0.382 (\pm {0.027}^{\circ})$
$R \to S$	$0.55 (\pm 0.028)$	0.578（±0.006）	$0.578 (\pm 0.006)$	$0.571 (\pm 0.006)$	$0.571 (\pm 0.012)$	$0.572 (\pm 0.005)$	$0.525 (\pm 0.088)$	$0.583 (\pm 0.016$	$0.588 (\pm 0.033)$
平均值	$0.499 (\pm 0.026)$	$0.515 (\pm 0.027)$	$0.515 (\pm 0.027)$	$0.508 (\pm 0.028)$	$0.507 (\pm 0.015)$	$0.493 (\pm 0.031)$	$0.498 (\pm 0.051)$	$0.519 (\pm 0.025)$	$0.541 (\pm 0.034)$

Task	DDC
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$R \to C$	$0.555 (\pm 0.045)$	0.555(±0.013)	$0.555 (\pm 0.013)$	$0.549 (\pm 0.008)$	$0.552 (\pm 0.009)$	$0.546 (\pm 0.013)$	$0.546 (\pm 0.013)$	$0.568 (\pm 0.025)$	$0.585 (\pm 0.076)$
$R \to I$	$0.374 (\pm 0.014)$	$0.391 (\pm 0.003)$	$0.391 (\pm 0.003)$	$0.38 (\pm 0.014)$	$0.398 (\pm 0.024)$	$0.379 (\pm 0.009)$	$0.379 (\pm 0.009)$	$0.385 (\pm 0.008)$	$0.416 (\pm 0.014)$
$R \to P$	$0.709 (\pm 0.025)$	$0.714 (\pm 0.0)$	$0.714 (\pm 0.0)$	$0.712 (\pm 0.004)$	$0.709 (\pm 0.004)$	$0.713 (\pm 0.006)$	0.718(±0.003	$0.715 (\pm 0.002)$	$0.717 (\pm 0.004)$
$R \to O$	$0.334 (\pm 0.002)$	$0.331 (\pm 0.003)$	$0.331 (\pm 0.003$	$0.333 (\pm 0.0)$	$0.327 (\pm 0.018)$	$0.333 (\pm 0.005)$	$0.335 (\pm 0.002)$	$0.335 (\pm 0.003)$	$0.337 (\pm 0.004)$
$R \to S$	$0.576 (\pm 0.022)$	$0.571 (\pm 0.01)$	$0.571 (\pm 0.01)$	$0.578 (\pm 0.008)$	$0.582 (\pm 0.005)$	$0.582 (\pm 0.005)$	$0.582 (\pm 0.005)$	$0.579 \hat{i} \pm 0.007$	$0.585 (\pm 0.021)$
Avg.	$0.51 (\pm 0.021)$	$0.512 (\pm 0.005)$	$0.512 (\pm 0.005)$	$0.511 (\pm 0.007)$	$0.514 (\pm 0.012)$	$0.511 (\pm 0.008)$	$0.512 (\pm 0.006)$	$0.516 (\pm 0.009)$	$0.528 (\pm 0.024)$

任务	DDC
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$R \to C$	$0.555 (\pm 0.045)$	0.555(±0.013)	$0.555 (\pm 0.013)$	$0.549 (\pm 0.008)$	$0.552 (\pm 0.009)$	$0.546 (\pm 0.013)$	$0.546 (\pm 0.013)$	$0.568 (\pm 0.025)$	$0.585 (\pm 0.076)$
$R \to I$	$0.374 (\pm 0.014)$	$0.391 (\pm 0.003)$	$0.391 (\pm 0.003)$	$0.38 (\pm 0.014)$	$0.398 (\pm 0.024)$	$0.379 (\pm 0.009)$	$0.379 (\pm 0.009)$	$0.385 (\pm 0.008)$	$0.416 (\pm 0.014)$
$R \to P$	$0.709 (\pm 0.025)$	$0.714 (\pm 0.0)$	$0.714 (\pm 0.0)$	$0.712 (\pm 0.004)$	$0.709 (\pm 0.004)$	$0.713 (\pm 0.006)$	0.718(±0.003)	$0.715 (\pm 0.002)$	$0.717 (\pm 0.004)$
$R \to O$	$0.334 (\pm 0.002)$	$0.331 (\pm 0.003)$	$0.331 (\pm 0.003$	$0.333 (\pm 0.0)$	$0.327 (\pm 0.018)$	$0.333 (\pm 0.005)$	$0.335 (\pm 0.002)$	$0.335 (\pm 0.003)$	$0.337 (\pm 0.004)$
$R \to S$	$0.576 (\pm 0.022)$	$0.571 (\pm 0.01)$	$0.571 (\pm 0.01)$	$0.578 (\pm 0.008)$	$0.582 (\pm 0.005)$	$0.582 (\pm 0.005)$	$0.582 (\pm 0.005)$	$0.579 \hat{i} \pm 0.007$	$0.585 (\pm 0.021)$
平均值	$0.51 (\pm 0.021)$	$0.512 (\pm 0.005)$	$0.512 (\pm 0.005)$	$0.511 (\pm 0.007)$	$0.514 (\pm 0.012)$	$0.511 (\pm 0.008)$	$0.512 (\pm 0.006)$	$0.516 (\pm 0.009)$	$0.528 (\pm 0.024)$

Task	CMD
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$R \to C$	$0.555 (\pm 0.042)$	0.598(±0.008)	$0.598 (\pm 0.008)$	$0.59 (\pm 0.008)$	$0.544 (\pm 0.021)$	$0.552 (\pm 0.031)$	$0.443 (\pm 0.067)$	$0.593 (\pm 0.009)$	$0.617 (\pm 0.025)$
$R \to I$	$0.377 (\pm 0.009)$	$0.368 (\pm 0.007)$	$0.368 (\pm 0.007)$	$0.371 (\pm 0.005)$	$0.395 (\pm 0.007)$	$0.379 (\pm 0.011)$	$0.379 (\pm 0.011)$	$0.379 (\pm 0.007)$	$0.389 (\pm 0.007)$
$R \to P$	$0.704 (\pm 0.02)$	0.741(±0.015	$0.741 (\pm 0.015)$	$0.733 (\pm 0.01)$	$0.723 (\pm 0.02)$	$0.732 (\pm 0.019)$	$0.736 (\pm 0.013)$	$0.74 (\pm 0.004)$	$0.734 (\pm 0.015)$
$R \to O$	$0.332 (\pm 0.015)$	$0.365 (\pm 0.005)$	$0.365 \hat{i} \pm 0.005$	$0.374 (\pm 0.003)$	$0.361 (\pm 0.036)$	$0.338 (\pm 0.059)$	$0.303 (\pm 0.061)$	$0.364 (\pm 0.007$	$0.378 (\pm 0.008)$
$R \to S$	$0.578 (\pm 0.022)$	$0.582 (\pm 0.01)$	$0.582 (\pm 0.01)$	$0.587 (\pm 0.005)$	$0.589 \dot{(} \pm 0.005 \dot{5}$	$0.588 (\pm 0.019)$	$0.591 (\pm 0.015)$	$0.592 (\pm 0.003$	$0.59 (\pm 0.003)$
Avg.	$0.509 (\pm 0.022)$	$0.531 (\pm 0.009)$	$0.531 (\pm 0.009)$	$0.531 (\pm 0.006)$	$0.522 (\pm 0.018)$	$0.518 (\pm 0.028)$	$0.49 (\pm 0.033)$	$0.533 (\pm 0.006)$	$0.542 (\pm 0.012)$

任务	命令
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$R \to C$	$0.555 (\pm 0.042)$	0.598（±0.008）	$0.598 (\pm 0.008)$	$0.59 (\pm 0.008)$	$0.544 (\pm 0.021)$	$0.552 (\pm 0.031)$	$0.443 (\pm 0.067)$	$0.593 (\pm 0.009)$	$0.617 (\pm 0.025)$
$R \to I$	$0.377 (\pm 0.009)$	$0.368 (\pm 0.007)$	$0.368 (\pm 0.007)$	$0.371 (\pm 0.005)$	$0.395 (\pm 0.007)$	$0.379 (\pm 0.011)$	$0.379 (\pm 0.011)$	$0.379 (\pm 0.007)$	$0.389 (\pm 0.007)$
$R \to P$	$0.704 (\pm 0.02)$	0.741（±0.015）	$0.741 (\pm 0.015)$	$0.733 (\pm 0.01)$	$0.723 (\pm 0.02)$	$0.732 (\pm 0.019)$	$0.736 (\pm 0.013)$	$0.74 (\pm 0.004)$	$0.734 (\pm 0.015)$
$R \to O$	$0.332 (\pm 0.015)$	$0.365 (\pm 0.005)$	$0.365 \hat{i} \pm 0.005$	$0.374 (\pm 0.003)$	$0.361 (\pm 0.036)$	$0.338 (\pm 0.059)$	$0.303 (\pm 0.061)$	$0.364 (\pm 0.007$	$0.378 (\pm 0.008)$
$R \to S$	$0.578 (\pm 0.022)$	$0.582 (\pm 0.01)$	$0.582 (\pm 0.01)$	$0.587 (\pm 0.005)$	$0.589 \dot{(} \pm 0.005 \dot{5}$	$0.588 (\pm 0.019)$	$0.591 (\pm 0.015)$	$0.592 (\pm 0.003$	$0.59 (\pm 0.003)$
平均值	$0.509 (\pm 0.022)$	$0.531 (\pm 0.009)$	$0.531 (\pm 0.009)$	$0.531 (\pm 0.006)$	$0.522 (\pm 0.018)$	$0.518 (\pm 0.028)$	$0.49 (\pm 0.033)$	$0.533 (\pm 0.006)$	$0.542 (\pm 0.012)$

Task	MMDA
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$R \to C$	$0.555 (\pm 0.042)$	0.587(±0.009)	$0.587 (\pm 0.009)$	$0.571 (\pm 0.013)$	$0.56 (\pm 0.019)$	$0.544 (\pm 0.033)$	$0.544 (\pm 0.033)$	$0.585 (\pm 0.009)$	$0.585 (\pm 0.009)$
$R \to I$	$0.377 (\pm 0.009)$	$0.379 (\pm 0.007)$	$0.379 (\pm 0.007)$	$0.376 (\pm 0.012)$	$0.389 (\pm 0.008)$	$0.389 (\pm 0.021)$	$0.389 (\pm 0.021$	$0.373 (\pm 0.003)$	$0.415 (\pm 0.016)$
$R \to P$	$0.704 (\pm 0.02)$	$0.714 (\pm 0.014)$	$0.714 (\pm 0.014)$	$0.714 (\pm 0.011)$	0.717(±0.014	$0.713 (\pm 0.003)$	$0.721 (\pm 0.015)$	$0.723 (\pm 0.004)$	$0.735 (\pm 0.007)$
$R \to O$	$0.332 (\pm 0.015)$	$0.352 (\pm 0.023)$	$0.352 (\pm 0.023)$	$0.372 (\pm 0.006)$	$0.335 (\pm 0.018)$	$0.353 (\pm 0.025)$	$0.38 (\pm 0.035)$	$0.362 (\pm 0.009)$	$0.392 (\pm 0.018)$
$R \to S$	$0.578 (\pm 0.022)$	$0.581 (\pm 0.011)$	$0.581 (\pm 0.011)$	$0.583 (\pm 0.013)$	0.592(±0.001	$0.587 (\pm 0.01)$	$0.587 (\pm 0.01)$	$0.593 (\pm 0.009)$	$0.605 (\pm 0.008)$
Avg.	$0.509 (\pm 0.022)$	$0.522 (\pm 0.013)$	$0.522 (\pm 0.013)$	$0.523 (\pm 0.011)$	$0.519 (\pm 0.013)$	$0.517 (\pm 0.018)$	$0.524 (\pm 0.023)$	$0.527 (\pm 0.007)$	$0.546 (\pm 0.012)$

任务	MMDA
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$R \to C$	$0.555 (\pm 0.042)$	0.587(±0.009)	$0.587 (\pm 0.009)$	$0.571 (\pm 0.013)$	$0.56 (\pm 0.019)$	$0.544 (\pm 0.033)$	$0.544 (\pm 0.033)$	$0.585 (\pm 0.009)$	$0.585 (\pm 0.009)$
$R \to I$	$0.377 (\pm 0.009)$	$0.379 (\pm 0.007)$	$0.379 (\pm 0.007)$	$0.376 (\pm 0.012)$	$0.389 (\pm 0.008)$	$0.389 (\pm 0.021)$	$0.389 (\pm 0.021$	$0.373 (\pm 0.003)$	$0.415 (\pm 0.016)$
$R \to P$	$0.704 (\pm 0.02)$	$0.714 (\pm 0.014)$	$0.714 (\pm 0.014)$	$0.714 (\pm 0.011)$	0.717(±0.014)	$0.713 (\pm 0.003)$	$0.721 (\pm 0.015)$	$0.723 (\pm 0.004)$	$0.735 (\pm 0.007)$
$R \to O$	$0.332 (\pm 0.015)$	$0.352 (\pm 0.023)$	$0.352 (\pm 0.023)$	$0.372 (\pm 0.006)$	$0.335 (\pm 0.018)$	$0.353 (\pm 0.025)$	$0.38 (\pm 0.035)$	$0.362 (\pm 0.009)$	$0.392 (\pm 0.018)$
$R \to S$	$0.578 (\pm 0.022)$	$0.581 (\pm 0.011)$	$0.581 (\pm 0.011)$	$0.583 (\pm 0.013)$	0.592(±0.001)	$0.587 (\pm 0.01)$	$0.587 (\pm 0.01)$	$0.593 (\pm 0.009)$	$0.605 (\pm 0.008)$
平均值	$0.509 (\pm 0.022)$	$0.522 (\pm 0.013)$	$0.522 (\pm 0.013)$	$0.523 (\pm 0.011)$	$0.519 (\pm 0.013)$	$0.517 (\pm 0.018)$	$0.524 (\pm 0.023)$	$0.527 (\pm 0.007)$	$0.546 (\pm 0.012)$

Task	CoDATS
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$R \to C$	$0.538 (\pm 0.072)$	$0.585 (\pm 0.025)$	$0.585 (\pm 0.025)$	$0.607 (\pm 0.028)$	$0.585 (\pm 0.013)$	$0.555 (\pm 0.031)$	$0.626 (\pm 0.058)$	$0.577 (\pm 0.026)$	$0.617 (\pm 0.058)$
$R \to I$	$0.365 (\pm 0.037)$	$0.371 (\pm 0.024)$	$0.371 (\pm 0.024)$	$0.367 (\pm 0.016)$	$0.38 (\pm 0.02)$	$0.373 (\pm 0.041)$	$0.363 (\pm 0.056)$	$0.371 (\pm 0.012)$	$0.406 (\pm 0.021)$
$R \to P$	$0.689 (\pm 0.024)$	$0.735 (\pm 0.01)$	$0.735 (\pm 0.01)$	$0.734 (\pm 0.015)$	$0.726 (\pm 0.014)$	$0.689 (\pm 0.043)$	$0.689 (\pm 0.043)$	$0.743 (\pm 0.016$	$0.732 (\pm 0.016)$
$R \to O$	$0.322 (\pm 0.022)$	$0.356 (\pm 0.02)$	$0.356 (\pm 0.02)$	$0.358 (\pm 0.017)$	$0.34 (\pm 0.017)$	$0.384 (\pm 0.012)$	$0.348 (\pm 0.044)$	$0.364 (\pm 0.012)$	$0.417 (\pm 0.025)$
$R \to S$	$0.597 (\pm 0.004)$	$0.619 (\pm 0.013)$	$0.619 (\pm 0.013)$	$0.616 (\pm 0.018)$	$0.593 (\pm 0.01)$	$0.592 (\pm 0.013)$	$0.592 (\pm 0.013)$	$0.622 (\pm 0.009)$	$0.627 (\pm 0.008)$
Avg.	$0.502 (\pm 0.032)$	$0.533 (\pm 0.018)$	$0.533 (\pm 0.018)$	$0.536 (\pm 0.019)$	$0.525 (\pm 0.015)$	$0.518 (\pm 0.028)$	$0.524 (\pm 0.043)$	$0.536 (\pm 0.015)$	$0.56 (\pm 0.026)$

任务	CoDATS
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$R \to C$	$0.538 (\pm 0.072)$	$0.585 (\pm 0.025)$	$0.585 (\pm 0.025)$	$0.607 (\pm 0.028)$	$0.585 (\pm 0.013)$	$0.555 (\pm 0.031)$	$0.626 (\pm 0.058)$	$0.577 (\pm 0.026)$	$0.617 (\pm 0.058)$
$R \to I$	$0.365 (\pm 0.037)$	$0.371 (\pm 0.024)$	$0.371 (\pm 0.024)$	$0.367 (\pm 0.016)$	$0.38 (\pm 0.02)$	$0.373 (\pm 0.041)$	$0.363 (\pm 0.056)$	$0.371 (\pm 0.012)$	$0.406 (\pm 0.021)$
$R \to P$	$0.689 (\pm 0.024)$	$0.735 (\pm 0.01)$	$0.735 (\pm 0.01)$	$0.734 (\pm 0.015)$	$0.726 (\pm 0.014)$	$0.689 (\pm 0.043)$	$0.689 (\pm 0.043)$	$0.743 (\pm 0.016$	$0.732 (\pm 0.016)$
$R \to O$	$0.322 (\pm 0.022)$	$0.356 (\pm 0.02)$	$0.356 (\pm 0.02)$	$0.358 (\pm 0.017)$	$0.34 (\pm 0.017)$	$0.384 (\pm 0.012)$	$0.348 (\pm 0.044)$	$0.364 (\pm 0.012)$	$0.417 (\pm 0.025)$
$R \to S$	$0.597 (\pm 0.004)$	$0.619 (\pm 0.013)$	$0.619 (\pm 0.013)$	$0.616 (\pm 0.018)$	$0.593 (\pm 0.01)$	$0.592 (\pm 0.013)$	$0.592 (\pm 0.013)$	$0.622 (\pm 0.009)$	$0.627 (\pm 0.008)$
平均值	$0.502 (\pm 0.032)$	$0.533 (\pm 0.018)$	$0.533 (\pm 0.018)$	$0.536 (\pm 0.019)$	$0.525 (\pm 0.015)$	$0.518 (\pm 0.028)$	$0.524 (\pm 0.043)$	$0.536 (\pm 0.015)$	$0.56 (\pm 0.026)$

Task	Deep-Coral
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$R \to C$	$0.555 (\pm 0.045)$	$0.585 (\pm 0.017)$	$0.585 (\pm 0.017)$	$0.582 (\pm 0.014)$	$0.557 (\pm 0.0)$	$0.536 (\pm 0.025)$	$0.568 (\pm 0.013)$	$0.59 (\pm 0.022)$	$0.601 (\pm 0.056)$
$R \to I$	$0.376 (\pm 0.012)$	$0.37 (\pm 0.01)$	$0.37 (\pm 0.01)$	$0.368 (\pm 0.01)$	0.38(±0.005	$0.368 (\pm 0.003)$	$0.368 (\pm 0.003$	$0.367 (\pm 0.012)$	$0.386 (\pm 0.007)$
$R \to P$	$0.711 (\pm 0.021)$	0.726(±0.015)	$0.726 (\pm 0.015)$	$0.723 (\pm 0.007)$	$0.714 (\pm 0.007)$	$0.722 (\pm 0.007)$	$0.722 (\pm 0.012)$	$0.738 (\pm 0.012$	$0.734 (\pm 0.012)$
$R \to Q$	$0.316 (\pm 0.041)$	$0.383 (\pm 0.003)$	$0.383 (\pm 0.003)$	0.386(±0.004	$0.365 (\pm 0.008)$	$0.378 (\pm 0.011)$	$0.375 (\pm 0.033)$	$0.38 (\pm 0.002)$	$0.388 (\pm 0.002)$
$R \to S$	$0.569 (\pm 0.02)$	$0.606 (\pm 0.009)$	$0.606 (\pm 0.009)$	$0.606 (\pm 0.005)$	$0.581 (\pm 0.007)$	$0.588 (\pm 0.008)$	$0.588 (\pm 0.008)$	$0.622 (\pm 0.001$	$0.658 (\pm 0.035)$
Avg.	$0.505 (\pm 0.028)$	$0.534 (\pm 0.011)$	$0.534 (\pm 0.011)$	$0.533 (\pm 0.008)$	$0.52 (\pm 0.005)$	$0.518 (\pm 0.011)$	$0.524 (\pm 0.014)$	$0.539 (\pm 0.01)$	$0.554 (\pm 0.022)$

任务	深度珊瑚
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$R \to C$	$0.555 (\pm 0.045)$	$0.585 (\pm 0.017)$	$0.585 (\pm 0.017)$	$0.582 (\pm 0.014)$	$0.557 (\pm 0.0)$	$0.536 (\pm 0.025)$	$0.568 (\pm 0.013)$	$0.59 (\pm 0.022)$	$0.601 (\pm 0.056)$
$R \to I$	$0.376 (\pm 0.012)$	$0.37 (\pm 0.01)$	$0.37 (\pm 0.01)$	$0.368 (\pm 0.01)$	0.38(±0.005)	$0.368 (\pm 0.003)$	$0.368 (\pm 0.003$	$0.367 (\pm 0.012)$	$0.386 (\pm 0.007)$
$R \to P$	$0.711 (\pm 0.021)$	0.726(±0.015)	$0.726 (\pm 0.015)$	$0.723 (\pm 0.007)$	$0.714 (\pm 0.007)$	$0.722 (\pm 0.007)$	$0.722 (\pm 0.012)$	$0.738 (\pm 0.012$	$0.734 (\pm 0.012)$
$R \to Q$	$0.316 (\pm 0.041)$	$0.383 (\pm 0.003)$	$0.383 (\pm 0.003)$	0.386(±0.004)	$0.365 (\pm 0.008)$	$0.378 (\pm 0.011)$	$0.375 (\pm 0.033)$	$0.38 (\pm 0.002)$	$0.388 (\pm 0.002)$
$R \to S$	$0.569 (\pm 0.02)$	$0.606 (\pm 0.009)$	$0.606 (\pm 0.009)$	$0.606 (\pm 0.005)$	$0.581 (\pm 0.007)$	$0.588 (\pm 0.008)$	$0.588 (\pm 0.008)$	$0.622 (\pm 0.001$	$0.658 (\pm 0.035)$
平均值	$0.505 (\pm 0.028)$	$0.534 (\pm 0.011)$	$0.534 (\pm 0.011)$	$0.533 (\pm 0.008)$	$0.52 (\pm 0.005)$	$0.518 (\pm 0.011)$	$0.524 (\pm 0.014)$	$0.539 (\pm 0.01)$	$0.554 (\pm 0.022)$

Task	CDAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$R \to C$	$0.557 (\pm 0.008)$	$0.587 (\pm 0.013)$	$0.587 (\pm 0.013)$	$0.593 (\pm 0.026)$	$0.563 (\pm 0.009)$	$0.552 (\pm 0.025)$	$0.574 (\pm 0.05)$	$0.598 (\pm 0.022)$	$0.615 (\pm 0.008)$
$R \to I$	$0.386 (\pm 0.01)$	$0.359 (\pm 0.005)$	$0.359 (\pm 0.005)$	$0.357 (\pm 0.009)$	$0.368 (\pm 0.005)$	$0.363 (\pm 0.04)$	$0.335 (\pm 0.016)$	$0.357 (\pm 0.009)$	$0.386 (\pm 0.01)$
$R \to P$	$0.709 (\pm 0.022)$	$0.732 (\pm 0.02)$	$0.732 (\pm 0.02)$	$0.734 (\pm 0.023)$	$0.71 (\pm 0.006)$	$0.72 (\pm 0.039)$	$0.723 (\pm 0.036)$	$0.731 (\pm 0.004$	$0.716 (\pm 0.023)$
$R \to O$	$0.342 (\pm 0.006)$	0.352(±0.024)	$0.352 (\pm 0.024)$	$0.349 (\pm 0.02)$	$0.347 (\pm 0.016)$	$0.356 (\pm 0.034)$	$0.38 (\pm 0.028$	$0.364 (\pm 0.013)$	$0.442 (\pm 0.114)$
$R \to S$	$0.576 (\pm 0.027)$	$0.588 (\pm 0.01)$	$0.588 (\pm 0.01)$	$0.593 (\pm 0.005)$	$0.596 (\pm 0.01)$	$0.574 (\pm 0.022)$	$0.568 (\pm 0.03)$	$0.61 (\pm 0.008)$	$0.651 (\pm 0.056)$
Avg.	$0.514 (\pm 0.015)$	$0.524 (\pm 0.014)$	$0.524 (\pm 0.014)$	$0.525 (\pm 0.017)$	$0.517 (\pm 0.009)$	$0.513 (\pm 0.032)$	$0.516 (\pm 0.032)$	$0.532 (\pm 0.011)$	$0.562 (\pm 0.042)$

任务	CDAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$R \to C$	$0.557 (\pm 0.008)$	$0.587 (\pm 0.013)$	$0.587 (\pm 0.013)$	$0.593 (\pm 0.026)$	$0.563 (\pm 0.009)$	$0.552 (\pm 0.025)$	$0.574 (\pm 0.05)$	$0.598 (\pm 0.022)$	$0.615 (\pm 0.008)$
$R \to I$	$0.386 (\pm 0.01)$	$0.359 (\pm 0.005)$	$0.359 (\pm 0.005)$	$0.357 (\pm 0.009)$	$0.368 (\pm 0.005)$	$0.363 (\pm 0.04)$	$0.335 (\pm 0.016)$	$0.357 (\pm 0.009)$	$0.386 (\pm 0.01)$
$R \to P$	$0.709 (\pm 0.022)$	$0.732 (\pm 0.02)$	$0.732 (\pm 0.02)$	$0.734 (\pm 0.023)$	$0.71 (\pm 0.006)$	$0.72 (\pm 0.039)$	$0.723 (\pm 0.036)$	$0.731 (\pm 0.004$	$0.716 (\pm 0.023)$
$R \to O$	$0.342 (\pm 0.006)$	0.352(±0.024)	$0.352 (\pm 0.024)$	$0.349 (\pm 0.02)$	$0.347 (\pm 0.016)$	$0.356 (\pm 0.034)$	$0.38 (\pm 0.028$	$0.364 (\pm 0.013)$	$0.442 (\pm 0.114)$
$R \to S$	$0.576 (\pm 0.027)$	$0.588 (\pm 0.01)$	$0.588 (\pm 0.01)$	$0.593 (\pm 0.005)$	$0.596 (\pm 0.01)$	$0.574 (\pm 0.022)$	$0.568 (\pm 0.03)$	$0.61 (\pm 0.008)$	$0.651 (\pm 0.056)$
平均值	$0.514 (\pm 0.015)$	$0.524 (\pm 0.014)$	$0.524 (\pm 0.014)$	$0.525 (\pm 0.017)$	$0.517 (\pm 0.009)$	$0.513 (\pm 0.032)$	$0.516 (\pm 0.032)$	$0.532 (\pm 0.011)$	$0.562 (\pm 0.042)$

Task	DANN
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$R \to C$	$0.519 (\pm 0.013)$	$0.626 (\pm 0.021)$	$0.626 (\pm 0.021)$	$0.623 (\pm 0.036)$	$0.557 (\pm 0.014)$	$0.582 (\pm 0.022)$	$0.56 (\pm 0.07)$	$0.607 (\pm 0.008)$	$0.631 (\pm 0.038)$
$R \to I$	$0.376 (\pm 0.016)$	$0.37 (\pm 0.013)$	$0.37 (\pm 0.013)$	$0.377 (\pm 0.013)$	$0.389 (\pm 0.032)$	$0.379 (\pm 0.036)$	$0.379 (\pm 0.036)$	$0.388 (\pm 0.019$	$0.4 (\pm 0.025)$
$R \to P$	$0.705 (\pm 0.011)$	$0.704 (\pm 0.013)$	$0.704 (\pm 0.013)$	$0.704 (\pm 0.011)$	$0.696 (\pm 0.006)$	$0.699 (\pm 0.006)$	$0.697 (\pm 0.014)$	$0.721 (\pm 0.008)$	$0.713 (\pm 0.008)$
$R \to O$	$0.3 (\pm 0.043)$	$0.377 (\pm 0.008)$	$0.377 (\pm 0.008)$	$0.38 (\pm 0.006$	$0.343 (\pm 0.004)$	$0.351 (\pm 0.048)$	$0.351 (\pm 0.048)$	$0.374 (\pm 0.003$	$0.392 (\pm 0.005)$
$R \to S$	$0.581 (\pm 0.014)$	$0.598 (\pm 0.023)$	$0.598 (\pm 0.023)$	$0.607 (\pm 0.016)$	$0.596 (\pm 0.01)$	$0.586 (\pm 0.015)$	$0.586 (\pm 0.015)$	$0.617 (\pm 0.005)$	$0.657 (\pm 0.015)$
Avg.	$0.496 (\pm 0.019)$	$0.535 (\pm 0.016)$	$0.535 (\pm 0.016)$	$0.538 (\pm 0.016)$	$0.516 (\pm 0.013)$	$0.519 (\pm 0.025)$	$0.515 (\pm 0.037)$	$0.541 (\pm 0.009)$	$0.559 (\pm 0.018)$

任务	DANN
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$R \to C$	$0.519 (\pm 0.013)$	$0.626 (\pm 0.021)$	$0.626 (\pm 0.021)$	$0.623 (\pm 0.036)$	$0.557 (\pm 0.014)$	$0.582 (\pm 0.022)$	$0.56 (\pm 0.07)$	$0.607 (\pm 0.008)$	$0.631 (\pm 0.038)$
$R \to I$	$0.376 (\pm 0.016)$	$0.37 (\pm 0.013)$	$0.37 (\pm 0.013)$	$0.377 (\pm 0.013)$	$0.389 (\pm 0.032)$	$0.379 (\pm 0.036)$	$0.379 (\pm 0.036)$	$0.388 (\pm 0.019$	$0.4 (\pm 0.025)$
$R \to P$	$0.705 (\pm 0.011)$	$0.704 (\pm 0.013)$	$0.704 (\pm 0.013)$	$0.704 (\pm 0.011)$	$0.696 (\pm 0.006)$	$0.699 (\pm 0.006)$	$0.697 (\pm 0.014)$	$0.721 (\pm 0.008)$	$0.713 (\pm 0.008)$
$R \to O$	$0.3 (\pm 0.043)$	$0.377 (\pm 0.008)$	$0.377 (\pm 0.008)$	$0.38 (\pm 0.006$	$0.343 (\pm 0.004)$	$0.351 (\pm 0.048)$	$0.351 (\pm 0.048)$	$0.374 (\pm 0.003$	$0.392 (\pm 0.005)$
$R \to S$	$0.581 (\pm 0.014)$	$0.598 (\pm 0.023)$	$0.598 (\pm 0.023)$	$0.607 (\pm 0.016)$	$0.596 (\pm 0.01)$	$0.586 (\pm 0.015)$	$0.586 (\pm 0.015)$	$0.617 (\pm 0.005)$	$0.657 (\pm 0.015)$
平均值	$0.496 (\pm 0.019)$	$0.535 (\pm 0.016)$	$0.535 (\pm 0.016)$	$0.538 (\pm 0.016)$	$0.516 (\pm 0.013)$	$0.519 (\pm 0.025)$	$0.515 (\pm 0.037)$	$0.541 (\pm 0.009)$	$0.559 (\pm 0.018)$

Task	DSAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$R \to C$	$0.555 (\pm 0.042)$	$0.637 (\pm 0.021)$	$0.637 (\pm 0.021)$	$0.639 (\pm 0.022)$	$0.566 (\pm 0.016)$	$0.557 (\pm 0.059)$	$0.571 (\pm 0.045)$	$0.639 (\pm 0.008)$	$0.648 (\pm 0.014)$
$R \to I$	$0.377 (\pm 0.009)$	$0.363 (\pm 0.01)$	$0.363 (\pm 0.01)$	$0.374 (\pm 0.015)$	$0.406 (\pm 0.005)$	$0.362 (\pm 0.034)$	$0.362 (\pm 0.034)$	$0.377 (\pm 0.016)$	$0.404 (\pm 0.005)$
$R \to P$	$0.705 (\pm 0.021)$	$0.712 (\pm 0.009)$	$0.712 (\pm 0.009)$	$0.714 (\pm 0.005)$	$0.72 (\pm 0.008)$	$0.716 (\pm 0.003)$	$0.719 (\pm 0.003)$	$0.719 (\pm 0.007$	$0.716 (\pm 0.008)$
$R \to O$	$0.332 (\pm 0.012)$	$0.364 (\pm 0.021)$	$0.364 (\pm 0.021)$	$0.38 (\pm 0.012)$	$0.34 (\pm 0.016)$	$0.342 (\pm 0.009)$	$0.437 (\pm 0.083)$	$0.372 (\pm 0.011)$	$0.478 (\pm 0.013)$
$R \to S$	$0.577 (\pm 0.026)$	$0.594 (\pm 0.004)$	$0.594 (\pm 0.004)$	$0.607 (\pm 0.005)$	$0.594 (\pm 0.003)$	$0.585 (\pm 0.004)$	$0.585 (\pm 0.004)$	$0.625 (\pm 0.016)$	$0.639 (\pm 0.052)$
Avg.	$0.509 (\pm 0.022)$	$0.534 (\pm 0.013)$	$0.534 (\pm 0.013)$	$0.543 (\pm 0.012)$	$0.525 (\pm 0.01)$	$0.513 (\pm 0.022)$	$0.535 (\pm 0.034)$	$0.546 (\pm 0.012)$	$0.577 (\pm 0.018)$

任务	DSAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$R \to C$	$0.555 (\pm 0.042)$	$0.637 (\pm 0.021)$	$0.637 (\pm 0.021)$	$0.639 (\pm 0.022)$	$0.566 (\pm 0.016)$	$0.557 (\pm 0.059)$	$0.571 (\pm 0.045)$	$0.639 (\pm 0.008)$	$0.648 (\pm 0.014)$
$R \to I$	$0.377 (\pm 0.009)$	$0.363 (\pm 0.01)$	$0.363 (\pm 0.01)$	$0.374 (\pm 0.015)$	$0.406 (\pm 0.005)$	$0.362 (\pm 0.034)$	$0.362 (\pm 0.034)$	$0.377 (\pm 0.016)$	$0.404 (\pm 0.005)$
$R \to P$	$0.705 (\pm 0.021)$	$0.712 (\pm 0.009)$	$0.712 (\pm 0.009)$	$0.714 (\pm 0.005)$	$0.72 (\pm 0.008)$	$0.716 (\pm 0.003)$	$0.719 (\pm 0.003)$	$0.719 (\pm 0.007$	$0.716 (\pm 0.008)$
$R \to O$	$0.332 (\pm 0.012)$	$0.364 (\pm 0.021)$	$0.364 (\pm 0.021)$	$0.38 (\pm 0.012)$	$0.34 (\pm 0.016)$	$0.342 (\pm 0.009)$	$0.437 (\pm 0.083)$	$0.372 (\pm 0.011)$	$0.478 (\pm 0.013)$
$R \to S$	$0.577 (\pm 0.026)$	$0.594 (\pm 0.004)$	$0.594 (\pm 0.004)$	$0.607 (\pm 0.005)$	$0.594 (\pm 0.003)$	$0.585 (\pm 0.004)$	$0.585 (\pm 0.004)$	$0.625 (\pm 0.016)$	$0.639 (\pm 0.052)$
平均值	$0.509 (\pm 0.022)$	$0.534 (\pm 0.013)$	$0.534 (\pm 0.013)$	$0.543 (\pm 0.012)$	$0.525 (\pm 0.01)$	$0.513 (\pm 0.022)$	$0.535 (\pm 0.034)$	$0.546 (\pm 0.012)$	$0.577 (\pm 0.018)$

Task	HoMM
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.565 (\pm 0.033)$	$0.557 (\pm 0.066)$	$0.557 (\pm 0.066)$	$0.553 (\pm 0.068)$	0.72(±0.052)	$0.694 (\pm 0.075)$	$0.488 (\pm 0.09)$	$0.719 (\pm 0.04)$	$0.654 (\pm 0.052)$
$12 \to 5$	$0.698 (\pm 0.081)$	$0.771 (\pm 0.014)$	$0.771 (\pm 0.014)$	$0.768 (\pm 0.008)$	0.788(±0.022	$0.776 (\pm 0.026)$	$0.776 (\pm 0.026)$	$0.762 (\pm 0.022)$	$0.811 (\pm 0.013)$
$16 \to 1$	$0.664 (\pm 0.068)$	0.702(±0.02)	$0.702 (\pm 0.02)$	$0.698 (\pm 0.025)$	$0.616 (\pm 0.017)$	$0.628 (\pm 0.017)$	$0.628 (\pm 0.017)$	$0.682 (\pm 0.008$	$0.729 (\pm 0.025)$
$7 \to 18$	$0.712 (\pm 0.024)$	$0.734 (\pm 0.021)$	$0.734 (\pm 0.021)$	0.736(±0.012	$0.678 (\pm 0.006)$	$0.729 (\pm 0.025)$	$0.734 (\pm 0.028)$	$0.74 (\pm 0.008)$	$0.755 (\pm 0.02)$
$9 \to 14$	$0.741 (\pm 0.031)$	$0.832 (\pm 0.008)$	$0.832 (\pm 0.008)$	$0.833 (\pm θ .0 t)$	$0.82 (\pm 0.024)$	$0.802 (\pm 0.025)$	$0.764 (\pm 0.07)$	$0.832 (\pm 0.01)$	$0.814 (\pm 0.01)$
Avg.	$0.676 (\pm 0.047)$	$0.719 (\pm 0.026)$	$0.719 (\pm 0.026)$	$0.718 (\pm 0.025)$	$0.724 (\pm 0.024)$	$0.726 (\pm 0.034)$	$0.678 (\pm 0.046)$	0.747(±0.018)	$0.753 (\pm 0.024)$

任务	HoMM
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.565 (\pm 0.033)$	$0.557 (\pm 0.066)$	$0.557 (\pm 0.066)$	$0.553 (\pm 0.068)$	0.72(±0.052)	$0.694 (\pm 0.075)$	$0.488 (\pm 0.09)$	$0.719 (\pm 0.04)$	$0.654 (\pm 0.052)$
$12 \to 5$	$0.698 (\pm 0.081)$	$0.771 (\pm 0.014)$	$0.771 (\pm 0.014)$	$0.768 (\pm 0.008)$	0.788(±0.022	$0.776 (\pm 0.026)$	$0.776 (\pm 0.026)$	$0.762 (\pm 0.022)$	$0.811 (\pm 0.013)$
$16 \to 1$	$0.664 (\pm 0.068)$	0.702(±0.02)	$0.702 (\pm 0.02)$	$0.698 (\pm 0.025)$	$0.616 (\pm 0.017)$	$0.628 (\pm 0.017)$	$0.628 (\pm 0.017)$	$0.682 (\pm 0.008$	$0.729 (\pm 0.025)$
$7 \to 18$	$0.712 (\pm 0.024)$	$0.734 (\pm 0.021)$	$0.734 (\pm 0.021)$	0.736(±0.012	$0.678 (\pm 0.006)$	$0.729 (\pm 0.025)$	$0.734 (\pm 0.028)$	$0.74 (\pm 0.008)$	$0.755 (\pm 0.02)$
$9 \to 14$	$0.741 (\pm 0.031)$	$0.832 (\pm 0.008)$	$0.832 (\pm 0.008)$	$0.833 (\pm θ .0 t)$	$0.82 (\pm 0.024)$	$0.802 (\pm 0.025)$	$0.764 (\pm 0.07)$	$0.832 (\pm 0.01)$	$0.814 (\pm 0.01)$
平均值	$0.676 (\pm 0.047)$	$0.719 (\pm 0.026)$	$0.719 (\pm 0.026)$	$0.718 (\pm 0.025)$	$0.724 (\pm 0.024)$	$0.726 (\pm 0.034)$	$0.678 (\pm 0.046)$	0.747(±0.018)	$0.753 (\pm 0.024)$

Task	AdvSKM
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.611 (\pm 0.005)$	$0.59 (\pm 0.02)$	$0.59 (\pm 0.02)$	$0.589 (\pm 0.025)$	$0.706 (\pm 0.043)$	$0.645 (\pm 0.038)$	$0.595 (\pm 0.035)$	$0.664 (\pm 0.05)$	$0.611 (\pm 0.043)$
$12 \to 5$	$0.723 (\pm 0.061)$	$0.75 (\pm 0.02)$	$0.75 (\pm 0.02)$	$0.747 (\pm 0.011)$	0.777(±0.024	$0.753 (\pm 0.015)$	$0.753 (\pm 0.015)$	$0.754 (\pm 0.027)$	$0.777 (\pm 0.051)$
$16 \to 1$	$0.558 (\pm 0.035)$	$0.667 (\pm 0.037)$	$0.667 (\pm 0.037)$	$0.671 (\pm 0.035$	$0.619 (\pm 0.043)$	$0.65 (\pm 0.076)$	$0.65 (\pm 0.076)$	$0.658 (\pm 0.03)$	$0.73 (\pm 0.033)$
$7 \to 18$	$0.651 (\pm 0.053)$	$0.72 (\pm 0.022)$	$0.72 (\pm 0.022)$	$0.72 (\pm 0.018)$	$0.669 (\pm 0.018)$	$0.706 (\pm 0.041)$	$0.681 (\pm 0.023$	0.707(±0.004	$0.723 (\pm 0.033)$
$9 \to 14$	$0.78 (\pm 0.027)$	$0.835 (\pm 0.016)$	$0.835 (\pm 0.016)$	$0.835 (\pm 0.016)$	$0.819 (\pm 0.032)$	$0.76 (\pm 0.059)$	$0.78 (\pm 0.027)$	$0.824 (\pm 0.014)$	$0.789 (\pm 0.016)$
Avg.	$0.665 (\pm 0.036)$	$0.712 (\pm 0.023)$	$0.712 (\pm 0.023)$	$0.712 (\pm 0.021)$	$0.718 (\pm 0.032)$	$0.703 (\pm 0.046)$	$0.692 (\pm 0.035)$	$0.722 (\pm 0.025)$	$0.726 (\pm 0.035)$

任务	AdvSKM
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.611 (\pm 0.005)$	$0.59 (\pm 0.02)$	$0.59 (\pm 0.02)$	$0.589 (\pm 0.025)$	$0.706 (\pm 0.043)$	$0.645 (\pm 0.038)$	$0.595 (\pm 0.035)$	$0.664 (\pm 0.05)$	$0.611 (\pm 0.043)$
$12 \to 5$	$0.723 (\pm 0.061)$	$0.75 (\pm 0.02)$	$0.75 (\pm 0.02)$	$0.747 (\pm 0.011)$	0.777(±0.024)	$0.753 (\pm 0.015)$	$0.753 (\pm 0.015)$	$0.754 (\pm 0.027)$	$0.777 (\pm 0.051)$
$16 \to 1$	$0.558 (\pm 0.035)$	$0.667 (\pm 0.037)$	$0.667 (\pm 0.037)$	$0.671 (\pm 0.035$	$0.619 (\pm 0.043)$	$0.65 (\pm 0.076)$	$0.65 (\pm 0.076)$	$0.658 (\pm 0.03)$	$0.73 (\pm 0.033)$
$7 \to 18$	$0.651 (\pm 0.053)$	$0.72 (\pm 0.022)$	$0.72 (\pm 0.022)$	$0.72 (\pm 0.018)$	$0.669 (\pm 0.018)$	$0.706 (\pm 0.041)$	$0.681 (\pm 0.023$	0.707(±0.004)	$0.723 (\pm 0.033)$
$9 \to 14$	$0.78 (\pm 0.027)$	$0.835 (\pm 0.016)$	$0.835 (\pm 0.016)$	$0.835 (\pm 0.016)$	$0.819 (\pm 0.032)$	$0.76 (\pm 0.059)$	$0.78 (\pm 0.027)$	$0.824 (\pm 0.014)$	$0.789 (\pm 0.016)$
平均值	$0.665 (\pm 0.036)$	$0.712 (\pm 0.023)$	$0.712 (\pm 0.023)$	$0.712 (\pm 0.021)$	$0.718 (\pm 0.032)$	$0.703 (\pm 0.046)$	$0.692 (\pm 0.035)$	$0.722 (\pm 0.025)$	$0.726 (\pm 0.035)$

Task	DIRT
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.582 (\pm 0.041)$	$0.493 (\pm 0.106)$	$0.493 (\pm 0.106)$	$0.509 (\pm 0.094)$	$0.625 (\pm 0.031)$	$0.447 (\pm 0.123)$	$0.487 (\pm 0.188)$	$0.54 (\pm 0.043)$	$0.582 (\pm 0.031)$
$12 \to 5$	$0.712 (\pm 0.08)$	$0.835 (\pm 0.015)$	$0.835 (\pm 0.015)$	$0.832 (\pm 0.017)$	0.87(±0.019)	$0.805 (\pm 0.061)$	$0.789 (\pm 0.105)$	$0.829 (\pm 0.02)$	$0.857 (\pm 0.019)$
$16 \to 1$	$0.526 (\pm 0.162)$	0.757(±0.025)	$0.757 (\pm 0.025)$	$0.755 (\pm 0.021)$	$0.671 (\pm 0.071)$	$0.618 (\pm 0.124)$	$0.618 (\pm 0.124)$	$0.746 (\pm 0.03)$	$0.797 (\pm 0.02)$
$7 \to 18$	$0.749 (\pm 0.041)$	$0.777 (\pm 0.018)$	$0.777 (\pm 0.018)$	$0.775 (\pm 0.005)$	$0.717 (\pm 0.044)$	$0.74 (\pm 0.069)$	$0.719 (\pm 0.072)$	$0.762 (\pm 0.0)$	$0.789 (\pm 0.02)$
$9 \to 14$	$0.711 (\pm 0.098)$	$0.862 (\pm 0.016)$	$0.862 (\pm 0.016)$	0.871(±0.014	$0.826 (\pm 0.027)$	$0.786 (\pm 0.071)$	$0.818 (\pm 0.045)$	$0.868 (\pm 0.012)$	$0.874 (\pm 0.022)$
Avg.	$0.656 (\pm 0.084)$	$0.745 (\pm 0.036)$	$0.745 (\pm 0.036)$	$0.748 (\pm 0.03)$	$0.742 (\pm 0.039)$	$0.679 (\pm 0.09)$	$0.686 (\pm 0.107)$	0.749(±0.021)	$0.78 (\pm 0.023)$

任务	DIRT
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.582 (\pm 0.041)$	$0.493 (\pm 0.106)$	$0.493 (\pm 0.106)$	$0.509 (\pm 0.094)$	$0.625 (\pm 0.031)$	$0.447 (\pm 0.123)$	$0.487 (\pm 0.188)$	$0.54 (\pm 0.043)$	$0.582 (\pm 0.031)$
$12 \to 5$	$0.712 (\pm 0.08)$	$0.835 (\pm 0.015)$	$0.835 (\pm 0.015)$	$0.832 (\pm 0.017)$	0.87(±0.019)	$0.805 (\pm 0.061)$	$0.789 (\pm 0.105)$	$0.829 (\pm 0.02)$	$0.857 (\pm 0.019)$
$16 \to 1$	$0.526 (\pm 0.162)$	0.757(±0.025)	$0.757 (\pm 0.025)$	$0.755 (\pm 0.021)$	$0.671 (\pm 0.071)$	$0.618 (\pm 0.124)$	$0.618 (\pm 0.124)$	$0.746 (\pm 0.03)$	$0.797 (\pm 0.02)$
$7 \to 18$	$0.749 (\pm 0.041)$	$0.777 (\pm 0.018)$	$0.777 (\pm 0.018)$	$0.775 (\pm 0.005)$	$0.717 (\pm 0.044)$	$0.74 (\pm 0.069)$	$0.719 (\pm 0.072)$	$0.762 (\pm 0.0)$	$0.789 (\pm 0.02)$
$9 \to 14$	$0.711 (\pm 0.098)$	$0.862 (\pm 0.016)$	$0.862 (\pm 0.016)$	0.871(±0.014)	$0.826 (\pm 0.027)$	$0.786 (\pm 0.071)$	$0.818 (\pm 0.045)$	$0.868 (\pm 0.012)$	$0.874 (\pm 0.022)$
平均值	$0.656 (\pm 0.084)$	$0.745 (\pm 0.036)$	$0.745 (\pm 0.036)$	$0.748 (\pm 0.03)$	$0.742 (\pm 0.039)$	$0.679 (\pm 0.09)$	$0.686 (\pm 0.107)$	0.749(±0.021)	$0.78 (\pm 0.023)$

Task	DDC
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.546 (\pm 0.033)$	$0.603 (\pm 0.029)$	$0.603 (\pm 0.029)$	$0.598 (\pm 0.027)$	$0.658 (\pm 0.06)$	$0.629 (\pm 0.059)$	$0.492 (\pm 0.032)$	$0.645 (\pm 0.095)$	$0.641 (\pm 0.06)$
$12 \to 5$	$0.695 (\pm 0.075)$	$0.751 (\pm 0.01)$	$0.751 (\pm 0.01)$	$0.749 (\pm 0.012)$	$0.779 (\pm 0.029)$	$0.716 (\pm 0.046)$	$0.716 (\pm 0.046)$	$0.755 (\pm 0.015)$	$0.783 (\pm 0.013)$
$16 \to 1$	$0.551 (\pm 0.091)$	$0.683 (\pm 0.013)$	$0.683 (\pm 0.013)$	$0.672 (\pm 0.005)$	$0.56 (\pm 0.028)$	$0.618 (\pm 0.021)$	$0.623 (\pm 0.012)$	$0.666 (\pm 0.017)$	$0.728 (\pm 0.051)$
$7 \to 18$	$0.699 (\pm 0.022)$	$0.729 (\pm 0.038)$	$0.729 (\pm 0.038)$	$0.717 (\pm 0.018)$	$0.674 (\pm 0.009)$	$0.724 (\pm 0.029)$	$0.712 (\pm 0.036)$	$0.733 (\pm 0.013)$	$0.743 (\pm 0.033)$
$9 \to 14$	$0.737 (\pm 0.035)$	$0.84 (\pm 0.014)$	$0.84 (\pm 0.014)$	$0.827 (\pm 0.025)$	$0.802 (\pm 0.014)$	$0.784 (\pm 0.01)$	$0.788 (\pm 0.042)$	$0.823 (\pm 0.016)$	$0.792 (\pm 0.014)$
Avg.	$0.646 (\pm 0.051)$	$0.721 (\pm 0.021)$	$0.721 (\pm 0.021)$	$0.712 (\pm 0.018)$	$0.695 (\pm 0.028)$	$0.694 (\pm 0.033)$	$0.666 (\pm 0.034)$	$0.724 (\pm 0.031)$	$0.737 (\pm 0.034)$

任务	DDC
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$0 \to 11$	$0.546 (\pm 0.033)$	$0.603 (\pm 0.029)$	$0.603 (\pm 0.029)$	$0.598 (\pm 0.027)$	$0.658 (\pm 0.06)$	$0.629 (\pm 0.059)$	$0.492 (\pm 0.032)$	$0.645 (\pm 0.095)$	$0.641 (\pm 0.06)$
$12 \to 5$	$0.695 (\pm 0.075)$	$0.751 (\pm 0.01)$	$0.751 (\pm 0.01)$	$0.749 (\pm 0.012)$	$0.779 (\pm 0.029)$	$0.716 (\pm 0.046)$	$0.716 (\pm 0.046)$	$0.755 (\pm 0.015)$	$0.783 (\pm 0.013)$
$16 \to 1$	$0.551 (\pm 0.091)$	$0.683 (\pm 0.013)$	$0.683 (\pm 0.013)$	$0.672 (\pm 0.005)$	$0.56 (\pm 0.028)$	$0.618 (\pm 0.021)$	$0.623 (\pm 0.012)$	$0.666 (\pm 0.017)$	$0.728 (\pm 0.051)$
$7 \to 18$	$0.699 (\pm 0.022)$	$0.729 (\pm 0.038)$	$0.729 (\pm 0.038)$	$0.717 (\pm 0.018)$	$0.674 (\pm 0.009)$	$0.724 (\pm 0.029)$	$0.712 (\pm 0.036)$	$0.733 (\pm 0.013)$	$0.743 (\pm 0.033)$
$9 \to 14$	$0.737 (\pm 0.035)$	$0.84 (\pm 0.014)$	$0.84 (\pm 0.014)$	$0.827 (\pm 0.025)$	$0.802 (\pm 0.014)$	$0.784 (\pm 0.01)$	$0.788 (\pm 0.042)$	$0.823 (\pm 0.016)$	$0.792 (\pm 0.014)$
平均	$0.646 (\pm 0.051)$	$0.721 (\pm 0.021)$	$0.721 (\pm 0.021)$	$0.712 (\pm 0.018)$	$0.695 (\pm 0.028)$	$0.694 (\pm 0.033)$	$0.666 (\pm 0.034)$	$0.724 (\pm 0.031)$	$0.737 (\pm 0.034)$

Task	CMD
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.536 (\pm 0.041)$	$0.564 (\pm 0.01)$	$0.564 (\pm 0.01)$	$0.551 (\pm 0.027)$	$0.655 (\pm 0.079)$	$0.607 (\pm 0.087)$	$0.505 (\pm 0.016)$	$0.581 (\pm 0.009)$	$0.611 (\pm 0.079)$
$12 \to 5$	$0.751 (\pm 0.032)$	$0.805 (\pm 0.012)$	$0.805 (\pm 0.012)$	$0.79 (\pm 0.016)$	$0.798 (\pm 0.016)$	$0.784 (\pm 0.03)$	$0.629 (\pm 0.044)$	$0.793 (\pm 0.014)$	$0.835 (\pm 0.02)$
$16 \to 1$	$0.557 (\pm 0.099)$	$0.747 (\pm 0.011)$	$0.747 (\pm 0.011)$	$0.701 (\pm 0.034)$	$0.624 (\pm 0.018)$	$0.678 (\pm 0.005)$	$0.665 (\pm 0.018)$	$0.69 (\pm 0.015)$	$0.749 (\pm 0.019)$
$7 \to 18$	$0.681 (\pm 0.047)$	$0.715 (\pm 0.017)$	$0.715 (\pm 0.017)$	0.727(±0.004)	$0.669 (\pm 0.019)$	$0.741 (\pm 0.005)$	$0.728 (\pm 0.025)$	$0.738 (\pm 0.004)$	$0.75 (\pm 0.026)$
$9 \to 14$	$0.74 (\pm 0.07)$	$0.85 (\pm 0.008)$	$0.85 (\pm 0.008)$	$0.848 (\pm 0.014)$	$0.797 (\pm 0.021)$	$0.772 (\pm 0.064)$	$0.672 (\pm 0.229)$	$0.841 (\pm 0.006$	$0.841 (\pm 0.008)$
Avg.	$0.653 (\pm 0.058)$	$0.736 (\pm 0.012)$	$0.736 (\pm 0.012)$	$0.723 (\pm 0.019)$	$0.709 (\pm 0.031)$	$0.716 (\pm 0.038)$	$0.64 (\pm 0.066)$	$0.729 (\pm 0.01)$	$0.757 (\pm 0.03)$

任务	CMD
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.536 (\pm 0.041)$	$0.564 (\pm 0.01)$	$0.564 (\pm 0.01)$	$0.551 (\pm 0.027)$	$0.655 (\pm 0.079)$	$0.607 (\pm 0.087)$	$0.505 (\pm 0.016)$	$0.581 (\pm 0.009)$	$0.611 (\pm 0.079)$
$12 \to 5$	$0.751 (\pm 0.032)$	$0.805 (\pm 0.012)$	$0.805 (\pm 0.012)$	$0.79 (\pm 0.016)$	$0.798 (\pm 0.016)$	$0.784 (\pm 0.03)$	$0.629 (\pm 0.044)$	$0.793 (\pm 0.014)$	$0.835 (\pm 0.02)$
$16 \to 1$	$0.557 (\pm 0.099)$	$0.747 (\pm 0.011)$	$0.747 (\pm 0.011)$	$0.701 (\pm 0.034)$	$0.624 (\pm 0.018)$	$0.678 (\pm 0.005)$	$0.665 (\pm 0.018)$	$0.69 (\pm 0.015)$	$0.749 (\pm 0.019)$
$7 \to 18$	$0.681 (\pm 0.047)$	$0.715 (\pm 0.017)$	$0.715 (\pm 0.017)$	0.727(±0.004)	$0.669 (\pm 0.019)$	$0.741 (\pm 0.005)$	$0.728 (\pm 0.025)$	$0.738 (\pm 0.004)$	$0.75 (\pm 0.026)$
$9 \to 14$	$0.74 (\pm 0.07)$	$0.85 (\pm 0.008)$	$0.85 (\pm 0.008)$	$0.848 (\pm 0.014)$	$0.797 (\pm 0.021)$	$0.772 (\pm 0.064)$	$0.672 (\pm 0.229)$	$0.841 (\pm 0.006$	$0.841 (\pm 0.008)$
平均值	$0.653 (\pm 0.058)$	$0.736 (\pm 0.012)$	$0.736 (\pm 0.012)$	$0.723 (\pm 0.019)$	$0.709 (\pm 0.031)$	$0.716 (\pm 0.038)$	$0.64 (\pm 0.066)$	$0.729 (\pm 0.01)$	$0.757 (\pm 0.03)$

Task	MMIDA
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.538 (\pm 0.032)$	$0.52 (\pm 0.01)$	$0.52 (\pm 0.01)$	$0.523 (\pm 0.007)$	$0.654 (\pm 0.025)$	$0.596 (\pm 0.086)$	$0.405 (\pm 0.099)$	$0.612 (\pm 0.037)$	$0.615 (\pm 0.025)$
$12 \to 5$	$0.749 (\pm 0.034)$	$0.797 (\pm 0.016)$	$0.797 (\pm 0.016)$	$0.79 (\pm 0.025)$	$0.798 (\pm 0.006)$	$0.698 (\pm 0.079)$	$0.747 (\pm 0.076)$	$0.793 (\pm 0.01)$	$0.823 (\pm 0.025)$
$16 \to 1$	$0.544 (\pm 0.116)$	$0.705 (\pm 0.015)$	$0.705 (\pm 0.015)$	$0.7 (\pm 0.022)$	$0.645 (\pm 0.032)$	$0.703 (\pm 0.02)$	$0.692 (\pm 0.008$	$0.7 (\pm 0.014)$	$0.739 (\pm 0.061)$
$7 \to 18$	$0.678 (\pm 0.046)$	$0.773 (\pm 0.01)$	$0.773 (\pm 0.01)$	$0.762 (\pm 0.0)$	$0.68 (\pm 0.051)$	$0.729 (\pm 0.094)$	$0.702 (\pm 0.082)$	$0.772 (\pm 0.005)$	$0.797 (\pm 0.004)$
$9 \to 14$	$0.742 (\pm 0.071)$	$0.84 (\pm 0.029)$	$0.84 (\pm 0.029)$	$0.839 (\pm 0.027)$	$0.793 (\pm 0.014)$	$0.792 (\pm 0.073)$	$0.755 (\pm 0.051)$	$0.85 (\pm 0.012)$	$0.835 (\pm 0.012)$
Avg.	$0.65 (\pm 0.06)$	$0.727 (\pm 0.016)$	$0.727 (\pm 0.016)$	$0.723 (\pm 0.016)$	$0.714 (\pm 0.026)$	$0.704 (\pm 0.07)$	$0.66 (\pm 0.063)$	$0.745 (\pm 0.016)$	$0.762 (\pm 0.025)$

任务	MMIDA
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.538 (\pm 0.032)$	$0.52 (\pm 0.01)$	$0.52 (\pm 0.01)$	$0.523 (\pm 0.007)$	$0.654 (\pm 0.025)$	$0.596 (\pm 0.086)$	$0.405 (\pm 0.099)$	$0.612 (\pm 0.037)$	$0.615 (\pm 0.025)$
$12 \to 5$	$0.749 (\pm 0.034)$	$0.797 (\pm 0.016)$	$0.797 (\pm 0.016)$	$0.79 (\pm 0.025)$	$0.798 (\pm 0.006)$	$0.698 (\pm 0.079)$	$0.747 (\pm 0.076)$	$0.793 (\pm 0.01)$	$0.823 (\pm 0.025)$
$16 \to 1$	$0.544 (\pm 0.116)$	$0.705 (\pm 0.015)$	$0.705 (\pm 0.015)$	$0.7 (\pm 0.022)$	$0.645 (\pm 0.032)$	$0.703 (\pm 0.02)$	$0.692 (\pm 0.008$	$0.7 (\pm 0.014)$	$0.739 (\pm 0.061)$
$7 \to 18$	$0.678 (\pm 0.046)$	$0.773 (\pm 0.01)$	$0.773 (\pm 0.01)$	$0.762 (\pm 0.0)$	$0.68 (\pm 0.051)$	$0.729 (\pm 0.094)$	$0.702 (\pm 0.082)$	$0.772 (\pm 0.005)$	$0.797 (\pm 0.004)$
$9 \to 14$	$0.742 (\pm 0.071)$	$0.84 (\pm 0.029)$	$0.84 (\pm 0.029)$	$0.839 (\pm 0.027)$	$0.793 (\pm 0.014)$	$0.792 (\pm 0.073)$	$0.755 (\pm 0.051)$	$0.85 (\pm 0.012)$	$0.835 (\pm 0.012)$
平均值	$0.65 (\pm 0.06)$	$0.727 (\pm 0.016)$	$0.727 (\pm 0.016)$	$0.723 (\pm 0.016)$	$0.714 (\pm 0.026)$	$0.704 (\pm 0.07)$	$0.66 (\pm 0.063)$	$0.745 (\pm 0.016)$	$0.762 (\pm 0.025)$

Task	CoDATS
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.568 (\pm 0.027)$	$0.565 (\pm 0.118)$	$0.565 (\pm 0.118)$	$0.556 (\pm 0.107)$	$0.625 (\pm 0.023)$	$0.539 (\pm 0.155)$	$0.564 (\pm 0.008)$	$0.602 (\pm 0.027)$	$0.6 (\pm 0.023)$
$12 \to 5$	$0.682 (\pm 0.043)$	$0.812 (\pm 0.014)$	$0.812 (\pm 0.014)$	$0.805 (\pm 0.014)$	$0.827 (\pm 0.012)$	$0.75 (\pm 0.024)$	$0.728 (\pm 0.018)$	$0.809 (\pm 0.007)$	$0.798 (\pm 0.012)$
$16 \to 1$	$0.574 (\pm 0.026)$	$0.732 (\pm 0.016)$	$0.732 (\pm 0.016)$	$0.729 (\pm 0.007)$	$0.642 (\pm 0.042)$	$0.633 (\pm 0.066)$	$0.676 (\pm 0.031)$	$0.729 (\pm 0.004)$	$0.735 (\pm 0.082)$
$7 \to 18$	$0.759 (\pm 0.039)$	$0.749 (\pm 0.023)$	$0.749 (\pm 0.023)$	$0.75 (\pm 0.02)$	$0.715 (\pm 0.014)$	$0.724 (\pm 0.008)$	$0.717 (\pm 0.092)$	$0.742 (\pm 0.01)$	$0.771 (\pm 0.02)$
$9 \to 14$	$0.779 (\pm 0.038)$	$0.837 (\pm 0.013)$	$0.837 (\pm 0.013)$	$0.839 (\pm 0.008$	$0.805 (\pm 0.01)$	$0.771 (\pm 0.026)$	$0.763 (\pm 0.006)$	0.84(±0.014)	$0.811 (\pm 0.014)$
Avg.	$0.672 (\pm 0.035)$	$0.739 (\pm 0.037)$	$0.739 (\pm 0.037)$	$0.736 (\pm 0.031)$	$0.723 (\pm 0.02)$	$0.683 (\pm 0.056)$	$0.69 (\pm 0.031)$	$0.744 (\pm 0.012)$	$0.743 (\pm 0.03)$

任务	CoDATS
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.568 (\pm 0.027)$	$0.565 (\pm 0.118)$	$0.565 (\pm 0.118)$	$0.556 (\pm 0.107)$	$0.625 (\pm 0.023)$	$0.539 (\pm 0.155)$	$0.564 (\pm 0.008)$	$0.602 (\pm 0.027)$	$0.6 (\pm 0.023)$
$12 \to 5$	$0.682 (\pm 0.043)$	$0.812 (\pm 0.014)$	$0.812 (\pm 0.014)$	$0.805 (\pm 0.014)$	$0.827 (\pm 0.012)$	$0.75 (\pm 0.024)$	$0.728 (\pm 0.018)$	$0.809 (\pm 0.007)$	$0.798 (\pm 0.012)$
$16 \to 1$	$0.574 (\pm 0.026)$	$0.732 (\pm 0.016)$	$0.732 (\pm 0.016)$	$0.729 (\pm 0.007)$	$0.642 (\pm 0.042)$	$0.633 (\pm 0.066)$	$0.676 (\pm 0.031)$	$0.729 (\pm 0.004)$	$0.735 (\pm 0.082)$
$7 \to 18$	$0.759 (\pm 0.039)$	$0.749 (\pm 0.023)$	$0.749 (\pm 0.023)$	$0.75 (\pm 0.02)$	$0.715 (\pm 0.014)$	$0.724 (\pm 0.008)$	$0.717 (\pm 0.092)$	$0.742 (\pm 0.01)$	$0.771 (\pm 0.02)$
$9 \to 14$	$0.779 (\pm 0.038)$	$0.837 (\pm 0.013)$	$0.837 (\pm 0.013)$	$0.839 (\pm 0.008$	$0.805 (\pm 0.01)$	$0.771 (\pm 0.026)$	$0.763 (\pm 0.006)$	0.84(±0.014)	$0.811 (\pm 0.014)$
平均值	$0.672 (\pm 0.035)$	$0.739 (\pm 0.037)$	$0.739 (\pm 0.037)$	$0.736 (\pm 0.031)$	$0.723 (\pm 0.02)$	$0.683 (\pm 0.056)$	$0.69 (\pm 0.031)$	$0.744 (\pm 0.012)$	$0.743 (\pm 0.03)$

Task	Deep-Coral
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	PWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.547 (\pm 0.032)$	$0.586 (\pm 0.028)$	$0.586 (\pm 0.028)$	$0.6 (\pm 0.029)$	$0.652 (\pm 0.058)$	$0.629 (\pm 0.063)$	$0.569 (\pm 0.11)$	$0.592 (\pm 0.028)$	$0.637 (\pm 0.058)$
$12 \to 5$	$0.694 (\pm 0.084)$	$0.75 (\pm 0.02)$	$0.75 (\pm 0.02)$	$0.745 (\pm 0.016)$	$0.781 (\pm 0.028$	$0.758 (\pm 0.036)$	$0.758 (\pm 0.036)$	$0.755 (\pm 0.018)$	$0.776 (\pm 0.028)$
$16 \to 1$	$0.545 (\pm {0.097}^{\circ})$	$0.68 (\pm 0.015)$	$0.68 (\pm 0.015)$	$0.671 (\pm 0.007)$	$0.564 (\pm 0.034)$	$0.605 (\pm 0.018)$	$0.553 (\pm 0.104)$	$0.662 (\pm 0.017)$	$0.728 (\pm 0.036)$
$7 \to 18$	$0.698 (\pm 0.022)$	$0.73 (\pm 0.042)$	$0.73 (\pm 0.042)$	$0.716 (\pm 0.02)$	$0.673 (\pm 0.006)$	$0.723 (\pm 0.027)$	$0.711 (\pm 0.034)$	$0.733 (\pm 0.013)$	$0.741 (\pm 0.016)$
$9 \to 14$	$0.733 (\pm 0.047)$	$0.836 (\pm 0.016)$	$0.836 (\pm 0.016)$	$0.826 (\pm 0.02)$	$0.802 (\pm 0.018)$	$0.786 (\pm 0.014)$	$0.784 (\pm 0.043)$	$0.82 (\pm 0.02)$	$0.792 (\pm 0.016)$
Avg.	$0.643 (\pm 0.056)$	$0.717 (\pm 0.024)$	$0.717 (\pm 0.024)$	$0.712 (\pm 0.018)$	$0.694 (\pm 0.029)$	0.7(±0.031)	$0.675 (\pm 0.065)$	$0.713 (\pm 0.019)$	$0.735 (\pm 0.031)$

任务	深度珊瑚
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	PWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.547 (\pm 0.032)$	$0.586 (\pm 0.028)$	$0.586 (\pm 0.028)$	$0.6 (\pm 0.029)$	$0.652 (\pm 0.058)$	$0.629 (\pm 0.063)$	$0.569 (\pm 0.11)$	$0.592 (\pm 0.028)$	$0.637 (\pm 0.058)$
$12 \to 5$	$0.694 (\pm 0.084)$	$0.75 (\pm 0.02)$	$0.75 (\pm 0.02)$	$0.745 (\pm 0.016)$	$0.781 (\pm 0.028$	$0.758 (\pm 0.036)$	$0.758 (\pm 0.036)$	$0.755 (\pm 0.018)$	$0.776 (\pm 0.028)$
$16 \to 1$	$0.545 (\pm {0.097}^{\circ})$	$0.68 (\pm 0.015)$	$0.68 (\pm 0.015)$	$0.671 (\pm 0.007)$	$0.564 (\pm 0.034)$	$0.605 (\pm 0.018)$	$0.553 (\pm 0.104)$	$0.662 (\pm 0.017)$	$0.728 (\pm 0.036)$
$7 \to 18$	$0.698 (\pm 0.022)$	$0.73 (\pm 0.042)$	$0.73 (\pm 0.042)$	$0.716 (\pm 0.02)$	$0.673 (\pm 0.006)$	$0.723 (\pm 0.027)$	$0.711 (\pm 0.034)$	$0.733 (\pm 0.013)$	$0.741 (\pm 0.016)$
$9 \to 14$	$0.733 (\pm 0.047)$	$0.836 (\pm 0.016)$	$0.836 (\pm 0.016)$	$0.826 (\pm 0.02)$	$0.802 (\pm 0.018)$	$0.786 (\pm 0.014)$	$0.784 (\pm 0.043)$	$0.82 (\pm 0.02)$	$0.792 (\pm 0.016)$
平均值	$0.643 (\pm 0.056)$	$0.717 (\pm 0.024)$	$0.717 (\pm 0.024)$	$0.712 (\pm 0.018)$	$0.694 (\pm 0.029)$	0.7(±0.031)	$0.675 (\pm 0.065)$	$0.713 (\pm 0.019)$	$0.735 (\pm 0.031)$

Task	CDAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.553 (\pm 0.024)$	$0.499 (\pm 0.065)$	$0.499 (\pm 0.065)$	$0.507 (\pm 0.079)$	$0.643 (\pm 0.053)$	$0.529 (\pm 0.111)$	$0.342 (\pm 0.024)$	$0.565 (\pm 0.078)$	$0.553 (\pm 0.053)$
$12 \to 5$	$0.658 (\pm 0.101)$	$0.844 (\pm 0.01)$	$0.844 (\pm 0.01)$	$0.84 (\pm 0.007)$	$0.829 (\pm 0.01)$	$0.792 (\pm 0.06)$	$0.671 (\pm 0.055)$	$0.835 (\pm 0.002)$	$0.85 (\pm 0.016)$
$16 \to 1$	$0.699 (\pm 0.077)$	$0.729 (\pm 0.023$	$0.729 (\pm 0.023)$	$0.726 (\pm 0.028)$	$0.659 (\pm 0.05)$	$0.626 (\pm 0.117)$	$0.682 (\pm 0.029)$	$0.706 (\pm 0.017)$	$0.764 (\pm 0.084)$
$7 \to 18$	$0.647 (\pm 0.045)$	$0.758 (\pm 0.017)$	$0.758 (\pm 0.017)$	$0.76 (\pm 0.016$	$0.704 (\pm 0.005)$	$0.73 (\pm 0.004)$	$0.754 (\pm 0.041)$	$0.773 (\pm 0.012)$	$0.794 (\pm 0.035)$
$9 \to 14$	$0.703 (\pm 0.041)$	$0.865 (\pm 0.018)$	$0.865 (\pm 0.018)$	$0.862 (\pm 0.028)$	$0.806 (\pm 0.032)$	$0.81 (\pm 0.053)$	$0.759 (\pm 0.041)$	$0.862 (\pm 0.018$	$0.832 (\pm 0.018)$
Avg.	$0.652 (\pm 0.058)$	$0.739 (\pm 0.027)$	$0.739 (\pm 0.027)$	$0.739 (\pm 0.032)$	$0.728 (\pm 0.03)$	$0.697 (\pm 0.069)$	$0.642 (\pm 0.038)$	$0.748 (\pm 0.025)$	$0.759 (\pm 0.041)$

任务	CDAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$0 \to 11$	$0.553 (\pm 0.024)$	$0.499 (\pm 0.065)$	$0.499 (\pm 0.065)$	$0.507 (\pm 0.079)$	$0.643 (\pm 0.053)$	$0.529 (\pm 0.111)$	$0.342 (\pm 0.024)$	$0.565 (\pm 0.078)$	$0.553 (\pm 0.053)$
$12 \to 5$	$0.658 (\pm 0.101)$	$0.844 (\pm 0.01)$	$0.844 (\pm 0.01)$	$0.84 (\pm 0.007)$	$0.829 (\pm 0.01)$	$0.792 (\pm 0.06)$	$0.671 (\pm 0.055)$	$0.835 (\pm 0.002)$	$0.85 (\pm 0.016)$
$16 \to 1$	$0.699 (\pm 0.077)$	$0.729 (\pm 0.023$	$0.729 (\pm 0.023)$	$0.726 (\pm 0.028)$	$0.659 (\pm 0.05)$	$0.626 (\pm 0.117)$	$0.682 (\pm 0.029)$	$0.706 (\pm 0.017)$	$0.764 (\pm 0.084)$
$7 \to 18$	$0.647 (\pm 0.045)$	$0.758 (\pm 0.017)$	$0.758 (\pm 0.017)$	$0.76 (\pm 0.016$	$0.704 (\pm 0.005)$	$0.73 (\pm 0.004)$	$0.754 (\pm 0.041)$	$0.773 (\pm 0.012)$	$0.794 (\pm 0.035)$
$9 \to 14$	$0.703 (\pm 0.041)$	$0.865 (\pm 0.018)$	$0.865 (\pm 0.018)$	$0.862 (\pm 0.028)$	$0.806 (\pm 0.032)$	$0.81 (\pm 0.053)$	$0.759 (\pm 0.041)$	$0.862 (\pm 0.018$	$0.832 (\pm 0.018)$
平均值	$0.652 (\pm 0.058)$	$0.739 (\pm 0.027)$	$0.739 (\pm 0.027)$	$0.739 (\pm 0.032)$	$0.728 (\pm 0.03)$	$0.697 (\pm 0.069)$	$0.642 (\pm 0.038)$	$0.748 (\pm 0.025)$	$0.759 (\pm 0.041)$

Task	DANN
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.551 (\pm 0.018)$	$0.602 (\pm 0.058)$	$0.602 (\pm 0.058)$	$0.602 (\pm 0.052)$	$0.672 (\pm 0.008)$	$0.594 (\pm 0.081)$	$0.495 (\pm 0.114)$	$0.622 (\pm 0.04)$	$0.574 (\pm 0.008)$
$12 \to 5$	$0.645 (\pm 0.075)$	$0.801 (\pm 0.014)$	$0.801 (\pm 0.014)$	$0.792 (\pm 0.014)$	$0.798 (\pm 0.018)$	$0.686 (\pm 0.051$	$0.686 (\pm 0.051)$	$0.784 (\pm 0.02)$	$0.819 (\pm 0.054)$
$16 \to 1$	$0.57 (\pm 0.136)$	$0.676 (\pm 0.002)$	$0.676 (\pm 0.002)$	$0.67 (\pm 0.008)$	$0.593 (\pm 0.025)$	$0.651 (\pm 0.055)$	$0.651 (\pm 0.055)$	0.661(±0.008	$0.747 (\pm 0.049)$
$7 \to 18$	$0.695 (\pm 0.037)$	$0.725 (\pm 0.006)$	$0.725 (\pm 0.006)$	$0.725 (\pm 0.014)$	$0.686 (\pm 0.013)$	$0.703 (\pm 0.034)$	$0.703 (\pm 0.034)$	$0.732 (\pm 0.011)$	$0.738 (\pm 0.018)$
$9 \to 14$	$0.746 (\pm 0.021)$	$0.814 (\pm 0.002)$	$0.814 (\pm 0.002)$	$0.815 (\pm 0.015)$	$0.822 (\pm 0.012$	$0.799 (\pm 0.039)$	$0.686 (\pm 0.085)$	$0.822 (\pm 0.01)$	$0.81 (\pm 0.01)$
Avg.	$0.641 (\pm 0.057)$	$0.723 (\pm 0.016)$	$0.723 (\pm 0.016)$	$0.721 (\pm 0.02)$	$0.714 (\pm 0.015)$	$0.687 (\pm 0.052)$	$0.644 (\pm 0.068)$	$0.724 (\pm 0.018)$	$0.738 (\pm 0.028)$

任务	DANN
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.551 (\pm 0.018)$	$0.602 (\pm 0.058)$	$0.602 (\pm 0.058)$	$0.602 (\pm 0.052)$	$0.672 (\pm 0.008)$	$0.594 (\pm 0.081)$	$0.495 (\pm 0.114)$	$0.622 (\pm 0.04)$	$0.574 (\pm 0.008)$
$12 \to 5$	$0.645 (\pm 0.075)$	$0.801 (\pm 0.014)$	$0.801 (\pm 0.014)$	$0.792 (\pm 0.014)$	$0.798 (\pm 0.018)$	$0.686 (\pm 0.051$	$0.686 (\pm 0.051)$	$0.784 (\pm 0.02)$	$0.819 (\pm 0.054)$
$16 \to 1$	$0.57 (\pm 0.136)$	$0.676 (\pm 0.002)$	$0.676 (\pm 0.002)$	$0.67 (\pm 0.008)$	$0.593 (\pm 0.025)$	$0.651 (\pm 0.055)$	$0.651 (\pm 0.055)$	0.661(±0.008	$0.747 (\pm 0.049)$
$7 \to 18$	$0.695 (\pm 0.037)$	$0.725 (\pm 0.006)$	$0.725 (\pm 0.006)$	$0.725 (\pm 0.014)$	$0.686 (\pm 0.013)$	$0.703 (\pm 0.034)$	$0.703 (\pm 0.034)$	$0.732 (\pm 0.011)$	$0.738 (\pm 0.018)$
$9 \to 14$	$0.746 (\pm 0.021)$	$0.814 (\pm 0.002)$	$0.814 (\pm 0.002)$	$0.815 (\pm 0.015)$	$0.822 (\pm 0.012$	$0.799 (\pm 0.039)$	$0.686 (\pm 0.085)$	$0.822 (\pm 0.01)$	$0.81 (\pm 0.01)$
平均值	$0.641 (\pm 0.057)$	$0.723 (\pm 0.016)$	$0.723 (\pm 0.016)$	$0.721 (\pm 0.02)$	$0.714 (\pm 0.015)$	$0.687 (\pm 0.052)$	$0.644 (\pm 0.068)$	$0.724 (\pm 0.018)$	$0.738 (\pm 0.028)$

Task	DSAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 11$	$0.547 (\pm 0.012)$	$0.539 (\pm 0.035)$	$0.539 (\pm 0.035)$	$0.527 (\pm 0.063)$	$0.668 (\pm 0.024)$	$0.59 (\pm 0.055)$	$0.311 (\pm 0.026)$	$0.629 (\pm 0.051)$	$0.603 (\pm 0.024)$
$12 \to 5$	$0.717 (\pm 0.086)$	$0.84 (\pm 0.02)$	$0.84 (\pm 0.02)$	$0.841 (\pm 0.022$	$0.837 (\pm 0.026)$	$0.822 (\pm 0.028)$	$0.699 (\pm 0.061$	$0.842 (\pm 0.018)$	$0.855 (\pm 0.007)$
$16 \to 1$	$0.697 (\pm 0.06)$	$0.755 (\pm 0.031)$	$0.755 (\pm 0.031)$	$0.722 (\pm 0.02)$	$0.612 (\pm 0.026)$	$0.617 (\pm 0.118)$	$0.617 (\pm 0.118$	$0.721 (\pm 0.014)$	$0.742 (\pm 0.031)$
$7 \to 18$	$0.684 (\pm 0.048)$	$0.734 (\pm 0.029)$	$0.734 (\pm 0.029)$	$0.741 (\pm 0.011)$	$0.71 (\pm 0.067)$	$0.749 (\pm 0.02)$	$0.745 (\pm 0.025)$	$0.764 (\pm 0.002)$	$0.768 (\pm 0.059)$
$9 \to 14$	$0.62 (\pm 0.04)$	$0.832 (\pm 0.024)$	$0.832 (\pm 0.024)$	$0.827 (\pm 0.018)$	$0.815 (\pm 0.018)$	$0.784 (\pm 0.046)$	$0.576 (\pm 0.154)$	$0.828 (\pm 0.021)$	$0.823 (\pm 0.024)$
Avg.	$0.653 (\pm 0.049)$	$0.74 (\pm 0.028)$	$0.74 (\pm 0.028)$	$0.732 (\pm 0.027)$	$0.728 (\pm 0.032)$	$0.712 (\pm 0.053)$	$0.589 (\pm 0.077)$	$0.757 (\pm 0.021)$	$0.758 (\pm 0.029)$

任务	DSAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 11$	$0.547 (\pm 0.012)$	$0.539 (\pm 0.035)$	$0.539 (\pm 0.035)$	$0.527 (\pm 0.063)$	$0.668 (\pm 0.024)$	$0.59 (\pm 0.055)$	$0.311 (\pm 0.026)$	$0.629 (\pm 0.051)$	$0.603 (\pm 0.024)$
$12 \to 5$	$0.717 (\pm 0.086)$	$0.84 (\pm 0.02)$	$0.84 (\pm 0.02)$	$0.841 (\pm 0.022$	$0.837 (\pm 0.026)$	$0.822 (\pm 0.028)$	$0.699 (\pm 0.061$	$0.842 (\pm 0.018)$	$0.855 (\pm 0.007)$
$16 \to 1$	$0.697 (\pm 0.06)$	$0.755 (\pm 0.031)$	$0.755 (\pm 0.031)$	$0.722 (\pm 0.02)$	$0.612 (\pm 0.026)$	$0.617 (\pm 0.118)$	$0.617 (\pm 0.118$	$0.721 (\pm 0.014)$	$0.742 (\pm 0.031)$
$7 \to 18$	$0.684 (\pm 0.048)$	$0.734 (\pm 0.029)$	$0.734 (\pm 0.029)$	$0.741 (\pm 0.011)$	$0.71 (\pm 0.067)$	$0.749 (\pm 0.02)$	$0.745 (\pm 0.025)$	$0.764 (\pm 0.002)$	$0.768 (\pm 0.059)$
$9 \to 14$	$0.62 (\pm 0.04)$	$0.832 (\pm 0.024)$	$0.832 (\pm 0.024)$	$0.827 (\pm 0.018)$	$0.815 (\pm 0.018)$	$0.784 (\pm 0.046)$	$0.576 (\pm 0.154)$	$0.828 (\pm 0.021)$	$0.823 (\pm 0.024)$
平均值	$0.653 (\pm 0.049)$	$0.74 (\pm 0.028)$	$0.74 (\pm 0.028)$	$0.732 (\pm 0.027)$	$0.728 (\pm 0.032)$	$0.712 (\pm 0.053)$	$0.589 (\pm 0.077)$	$0.757 (\pm 0.021)$	$0.758 (\pm 0.029)$

Task	HoMM
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.674 (\pm 0.012)$	$0.667 (\pm 0.036)$	$0.667 (\pm 0.036)$	$0.667 (\pm 0.036)$	$0.688 (\pm 0.0)$	$0.674 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.715 (\pm 0.012)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	1.0(±0.0)	$0.938 (\pm 0.031)$	$0.99 (\pm 0.018)$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.854 (\pm 0.036)$	0.896(±0.0)	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.889 (\pm 0.012)$	$0.854 (\pm 0.036)$	$0.854 (\pm 0.036)$	$0.903 (\pm 0.012)$	$0.938 (\pm 0.0)$
$7 \to 13$	$0.889 (\pm 0.052)$	0.91(±0.024)	$0.91 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.889 (\pm 0.052)$	$0.889 (\pm 0.052)$	$0.889 (\pm 0.052)$	$0.91 (\pm 0.024)$	$0.944 (\pm 0.024)$
$9 \to 18$	$0.493 (\pm 0.032)$	$0.618 (\pm 0.105)$	$0.618 (\pm 0.105)$	$0.618 (\pm 0.105)$	$0.514 (\pm 0.188)$	$0.639 (\pm 0.043)$	$0.583 (\pm 0.116)$	$0.632 (\pm 0.079)$	$0.715 (\pm 0.024)$
Avg.	$0.782 (\pm 0.026)$	$0.818 (\pm 0.033)$	$0.818 (\pm 0.033)$	$0.818 (\pm 0.033)$	$0.783 (\pm 0.057)$	$0.809 (\pm 0.032)$	$0.8 (\pm 0.043)$	$0.826 (\pm 0.023)$	$0.862 (\pm 0.012)$

任务	HoMM
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$12 \to 16$	$0.674 (\pm 0.012)$	$0.667 (\pm 0.036)$	$0.667 (\pm 0.036)$	$0.667 (\pm 0.036)$	$0.688 (\pm 0.0)$	$0.674 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.715 (\pm 0.012)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	1.0(±0.0)	$0.938 (\pm 0.031)$	$0.99 (\pm 0.018)$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.854 (\pm 0.036)$	0.896(±0.0)	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.889 (\pm 0.012)$	$0.854 (\pm 0.036)$	$0.854 (\pm 0.036)$	$0.903 (\pm 0.012)$	$0.938 (\pm 0.0)$
$7 \to 13$	$0.889 (\pm 0.052)$	0.91(±0.024)	$0.91 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.889 (\pm 0.052)$	$0.889 (\pm 0.052)$	$0.889 (\pm 0.052)$	$0.91 (\pm 0.024)$	$0.944 (\pm 0.024)$
$9 \to 18$	$0.493 (\pm 0.032)$	$0.618 (\pm 0.105)$	$0.618 (\pm 0.105)$	$0.618 (\pm 0.105)$	$0.514 (\pm 0.188)$	$0.639 (\pm 0.043)$	$0.583 (\pm 0.116)$	$0.632 (\pm 0.079)$	$0.715 (\pm 0.024)$
平均值	$0.782 (\pm 0.026)$	$0.818 (\pm 0.033)$	$0.818 (\pm 0.033)$	$0.818 (\pm 0.033)$	$0.783 (\pm 0.057)$	$0.809 (\pm 0.032)$	$0.8 (\pm 0.043)$	$0.826 (\pm 0.023)$	$0.862 (\pm 0.012)$

Task	AdvSKM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.59 (\pm 0.103)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.708 (\pm 0.021)$	$0.59 (\pm 0.103)$	$0.59 (\pm 0.103)$	$0.688 (\pm 0.0)$	$0.722 (\pm 0.012)$
$2 \to 11$	$0.979 (\pm 0.018)$	1.0(±0.0)	$1.0 (\pm 0.0)$	$1.0 (\pm 0.0)$	$0.854 (\pm 0.065)$	$0.896 (\pm 0.1)$	$0.917 (\pm 0.118)$	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.826 (\pm 0.084)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.889 (\pm 0.012)$	$0.826 (\pm 0.084)$	$0.826 (\pm 0.084)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$
$7 \to 13$	$0.833 (\pm 0.036)$	$0.91 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.917 (\pm 0.036)$	$0.896 (\pm 0.036)$	$0.833 (\pm 0.036)$	$0.833 (\pm 0.036)$	$0.903 (\pm 0.024)$	$0.924 (\pm 0.012)$
$9 \to 18$	$0.389 (\pm 0.151)$	$0.514 (\pm 0.079)$	$0.514 (\pm 0.079)$	$0.556 (\pm 0.064)$	$0.493 (\pm 0.064)$	$0.389 (\pm 0.151)$	$0.389 (\pm 0.151)$	$0.514 (\pm 0.024)$	$0.597 (\pm 0.052)$
Avg.	$0.724 (\pm 0.078)$	$0.8 (\pm 0.023)$	$0.8 (\pm 0.023)$	$0.81 (\pm 0.022)$	$0.768 (\pm 0.04)$	$0.707 (\pm 0.095)$	$0.711 (\pm 0.098)$	0.8(±0.01)	$0.828 (\pm 0.015)$

任务	AdvSKM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$12 \to 16$	$0.59 (\pm 0.103)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.708 (\pm 0.021)$	$0.59 (\pm 0.103)$	$0.59 (\pm 0.103)$	$0.688 (\pm 0.0)$	$0.722 (\pm 0.012)$
$2 \to 11$	$0.979 (\pm 0.018)$	1.0(±0.0)	$1.0 (\pm 0.0)$	$1.0 (\pm 0.0)$	$0.854 (\pm 0.065)$	$0.896 (\pm 0.1)$	$0.917 (\pm 0.118)$	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.826 (\pm 0.084)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.889 (\pm 0.012)$	$0.826 (\pm 0.084)$	$0.826 (\pm 0.084)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$
$7 \to 13$	$0.833 (\pm 0.036)$	$0.91 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.917 (\pm 0.036)$	$0.896 (\pm 0.036)$	$0.833 (\pm 0.036)$	$0.833 (\pm 0.036)$	$0.903 (\pm 0.024)$	$0.924 (\pm 0.012)$
$9 \to 18$	$0.389 (\pm 0.151)$	$0.514 (\pm 0.079)$	$0.514 (\pm 0.079)$	$0.556 (\pm 0.064)$	$0.493 (\pm 0.064)$	$0.389 (\pm 0.151)$	$0.389 (\pm 0.151)$	$0.514 (\pm 0.024)$	$0.597 (\pm 0.052)$
平均值	$0.724 (\pm 0.078)$	$0.8 (\pm 0.023)$	$0.8 (\pm 0.023)$	$0.81 (\pm 0.022)$	$0.768 (\pm 0.04)$	$0.707 (\pm 0.095)$	$0.711 (\pm 0.098)$	0.8(±0.01)	$0.828 (\pm 0.015)$

Task	DIRT
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.667 (\pm 0.075)$	$0.778 (\pm 0.012)$	$0.778 (\pm 0.012)$	$0.75 (\pm 0.0)$	$0.736 (\pm 0.012)$	$0.667 (\pm 0.075)$	$0.667 (\pm 0.075)$	$0.764 (\pm 0.032)$	$0.819 (\pm 0.064)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	$1.0 (\pm 0.0)$	$0.854 (\pm 0.079)$	1.0(±0.0)	$1.0 (\pm 0.0)$	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.903 (\pm 0.084)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.847 (\pm 0.06)$	$0.847 (\pm 0.06)$	$0.847 (\pm 0.06)$	0.938(±0.0)	$0.938 (\pm 0.0)$
$7 \to 13$	$0.833 (\pm 0.0)$	$0.958 (\pm 0.0)$	$0.958 (\pm 0.0)$	$0.958 (\pm 0.0)$	$0.924 (\pm 0.06)$	$0.854 (\pm 0.036)$	$0.861 (\pm 0.032)$	$0.958 (\pm 0.0)$	$0.958 (\pm 0.0)$
$9 \to 18$	$0.514 (\pm 0.024)$	$0.861 (\pm 0.122)$	$0.861 (\pm 0.122)$	$0.806 (\pm 0.032)$	$0.417 (\pm 0.075)$	$0.667 (\pm 0.225)$	$0.667 (\pm 0.225)$	$0.84 (\pm 0.048)$	$0.986 (\pm 0.012)$
Avg.	$0.783 (\pm 0.037)$	$0.907 (\pm 0.027)$	$0.907 (\pm 0.027)$	$0.89 (\pm 0.006)$	$0.756 (\pm 0.057)$	$0.807 (\pm 0.079)$	$0.808 (\pm 0.078)$	0.9(±0.016)	$0.94 (\pm 0.015)$

任务	DIRT
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$12 \to 16$	$0.667 (\pm 0.075)$	$0.778 (\pm 0.012)$	$0.778 (\pm 0.012)$	$0.75 (\pm 0.0)$	$0.736 (\pm 0.012)$	$0.667 (\pm 0.075)$	$0.667 (\pm 0.075)$	$0.764 (\pm 0.032)$	$0.819 (\pm 0.064)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	$1.0 (\pm 0.0)$	$0.854 (\pm 0.079)$	1.0(±0.0)	$1.0 (\pm 0.0)$	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.903 (\pm 0.084)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.847 (\pm 0.06)$	$0.847 (\pm 0.06)$	$0.847 (\pm 0.06)$	0.938(±0.0)	$0.938 (\pm 0.0)$
$7 \to 13$	$0.833 (\pm 0.0)$	$0.958 (\pm 0.0)$	$0.958 (\pm 0.0)$	$0.958 (\pm 0.0)$	$0.924 (\pm 0.06)$	$0.854 (\pm 0.036)$	$0.861 (\pm 0.032)$	$0.958 (\pm 0.0)$	$0.958 (\pm 0.0)$
$9 \to 18$	$0.514 (\pm 0.024)$	$0.861 (\pm 0.122)$	$0.861 (\pm 0.122)$	$0.806 (\pm 0.032)$	$0.417 (\pm 0.075)$	$0.667 (\pm 0.225)$	$0.667 (\pm 0.225)$	$0.84 (\pm 0.048)$	$0.986 (\pm 0.012)$
平均值	$0.783 (\pm 0.037)$	$0.907 (\pm 0.027)$	$0.907 (\pm 0.027)$	$0.89 (\pm 0.006)$	$0.756 (\pm 0.057)$	$0.807 (\pm 0.079)$	$0.808 (\pm 0.078)$	0.9(±0.016)	$0.94 (\pm 0.015)$

Task	DDC
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TR
$12 \to 16$	$0.681 (\pm 0.024)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.681 (\pm 0.024)$	$0.604 (\pm 0.108)$	$0.681 (\pm 0.012)$	$0.715 (\pm 0.073)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.74 (\pm 0.213)$	$0.74 (\pm 0.213)$	$0.802 (\pm 0.266)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.889 (\pm 0.012)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.882 (\pm 0.012)$	$0.889 (\pm 0.012)$	$0.896 (\pm 0.021)$	$0.896 (\pm 0.0)$	$0.889 (\pm 0.0)$
$7 \to 13$	$0.868 (\pm 0.012)$	$0.896 (\pm 0.036)$	$0.896 (\pm 0.036)$	$0.91 (\pm 0.032)$	$0.903 (\pm 0.024)$	$0.868 (\pm 0.012)$	$0.868 (\pm 0.012)$	$0.903 (\pm 0.024)$	$0.917 (\pm 0.021)$
$9 \to 18$	$0.514 (\pm 0.043)$	$0.562 (\pm 0.091)$	$0.562 (\pm 0.091)$	$0.569 (\pm 0.115)$	$0.583 (\pm 0.165)$	$0.444 (\pm 0.103)$	$0.5 (\pm 0.127)$	$0.542 (\pm 0.091$	$0.59 (\pm 0.189)$
Avg.	$0.79 (\pm 0.018)$	$0.807 (\pm 0.028)$	$0.807 (\pm 0.028)$	$0.81 (\pm 0.032)$	$0.756 (\pm 0.085)$	$0.724 (\pm 0.073)$	$0.734 (\pm 0.107)$	$0.804 (\pm 0.025)$	$0.822 (\pm 0.057)$

任务	DDC
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TR
$12 \to 16$	$0.681 (\pm 0.024)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.681 (\pm 0.024)$	$0.604 (\pm 0.108)$	$0.681 (\pm 0.012)$	$0.715 (\pm 0.073)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.74 (\pm 0.213)$	$0.74 (\pm 0.213)$	$0.802 (\pm 0.266)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.889 (\pm 0.012)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.882 (\pm 0.012)$	$0.889 (\pm 0.012)$	$0.896 (\pm 0.021)$	$0.896 (\pm 0.0)$	$0.889 (\pm 0.0)$
$7 \to 13$	$0.868 (\pm 0.012)$	$0.896 (\pm 0.036)$	$0.896 (\pm 0.036)$	$0.91 (\pm 0.032)$	$0.903 (\pm 0.024)$	$0.868 (\pm 0.012)$	$0.868 (\pm 0.012)$	$0.903 (\pm 0.024)$	$0.917 (\pm 0.021)$
$9 \to 18$	$0.514 (\pm 0.043)$	$0.562 (\pm 0.091)$	$0.562 (\pm 0.091)$	$0.569 (\pm 0.115)$	$0.583 (\pm 0.165)$	$0.444 (\pm 0.103)$	$0.5 (\pm 0.127)$	$0.542 (\pm 0.091$	$0.59 (\pm 0.189)$
平均值	$0.79 (\pm 0.018)$	$0.807 (\pm 0.028)$	$0.807 (\pm 0.028)$	$0.81 (\pm 0.032)$	$0.756 (\pm 0.085)$	$0.724 (\pm 0.073)$	$0.734 (\pm 0.107)$	$0.804 (\pm 0.025)$	$0.822 (\pm 0.057)$

Task	CMD
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.694 (\pm 0.012)$	$0.729 (\pm 0.0)$	$0.729 (\pm 0.0)$	$0.722 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.694 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.729 (\pm 0.0)$	$0.75 (\pm 0.036)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	1.0(±0.0)	$0.771 (\pm 0.036)$	$0.812 (\pm 0.272)$	$0.812 (\pm 0.272)$	1.0 $(\pm 0.0)$	$1.0 (\pm 0.0)$
$6 \to 23$	$0.896 (\pm 0.042)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.931 (\pm 0.012)$	$0.889 (\pm 0.012)$	$0.903 (\pm 0.032)$	$0.896 (\pm 0.042)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$
$7 \to 13$	$0.826 (\pm 0.073)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.91 (\pm 0.024)$	$0.826 (\pm 0.073)$	$0.826 (\pm 0.073)$	$0.938 (\pm 0.0)$	$0.944 (\pm 0.012)$
$9 \to 18$	$0.521 (\pm 0.095)$	$0.639 (\pm 0.043)$	$0.639 (\pm 0.043)$	$0.604 (\pm 0.055)$	$0.396 (\pm 0.11)$	$0.785 (\pm 0.146)$	$0.833 (\pm 0.163)$	$0.604 (\pm 0.075)$	$0.882 (\pm 0.079)$
Avg.	$0.788 (\pm 0.044)$	$0.849 (\pm 0.009)$	$0.849 (\pm 0.009)$	$0.839 (\pm 0.016)$	$0.731 (\pm 0.036)$	$0.804 (\pm 0.107)$	$0.812 (\pm 0.112)$	$0.842 (\pm 0.015)$	$0.903 (\pm 0.025)$

任务	命令
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$12 \to 16$	$0.694 (\pm 0.012)$	$0.729 (\pm 0.0)$	$0.729 (\pm 0.0)$	$0.722 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.694 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.729 (\pm 0.0)$	$0.75 (\pm 0.036)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	1.0(±0.0)	$0.771 (\pm 0.036)$	$0.812 (\pm 0.272)$	$0.812 (\pm 0.272)$	1.0 $(\pm 0.0)$	$1.0 (\pm 0.0)$
$6 \to 23$	$0.896 (\pm 0.042)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.931 (\pm 0.012)$	$0.889 (\pm 0.012)$	$0.903 (\pm 0.032)$	$0.896 (\pm 0.042)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$
$7 \to 13$	$0.826 (\pm 0.073)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.91 (\pm 0.024)$	$0.826 (\pm 0.073)$	$0.826 (\pm 0.073)$	$0.938 (\pm 0.0)$	$0.944 (\pm 0.012)$
$9 \to 18$	$0.521 (\pm 0.095)$	$0.639 (\pm 0.043)$	$0.639 (\pm 0.043)$	$0.604 (\pm 0.055)$	$0.396 (\pm 0.11)$	$0.785 (\pm 0.146)$	$0.833 (\pm 0.163)$	$0.604 (\pm 0.075)$	$0.882 (\pm 0.079)$
平均值	$0.788 (\pm 0.044)$	$0.849 (\pm 0.009)$	$0.849 (\pm 0.009)$	$0.839 (\pm 0.016)$	$0.731 (\pm 0.036)$	$0.804 (\pm 0.107)$	$0.812 (\pm 0.112)$	$0.842 (\pm 0.015)$	$0.903 (\pm 0.025)$

Task	MMDA
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.674 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.681 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.757 (\pm 0.043)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	$1.0 (\pm 0.0)$	$0.844 (\pm 0.125)$	$0.927 (\pm 0.065)$	$0.927 (\pm 0.065)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.847 (\pm 0.032)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.847 (\pm 0.067)$	$0.847 (\pm 0.032)$	$0.847 (\pm 0.032)$	$0.896 (\pm 0.0)$	$0.91 (\pm 0.024)$
$7 \to 13$	$0.882 (\pm 0.043)$	$0.903 (\pm 0.012)$	$0.903 (\pm 0.012)$	$0.917 (\pm 0.021)$	$0.924 (\pm 0.024)$	$0.882 (\pm 0.043)$	$0.882 (\pm 0.043)$	$0.931 (\pm 0.024)$	$0.944 (\pm 0.012)$
$9 \to 18$	$0.521 (\pm 0.072)$	$0.569 (\pm 0.139)$	$0.569 (\pm 0.139)$	$0.5 (\pm 0.104)$	$0.5 (\pm 0.095)$	$0.535 (\pm 0.064)$	$0.507 (\pm 0.024)$	$0.521 (\pm 0.091)$	$0.688 (\pm 0.075)$
Avg.	$0.785 (\pm 0.032)$	$0.812 (\pm 0.033)$	$0.812 (\pm 0.033)$	$0.8 (\pm 0.025)$	$0.759 (\pm 0.065)$	$0.773 (\pm 0.043)$	$0.767 (\pm 0.035)$	$0.807 (\pm 0.023)$	$0.86 (\pm 0.031)$

任务	MMDA
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$12 \to 16$	$0.674 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.681 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.688 (\pm 0.0)$	$0.757 (\pm 0.043)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	$1.0 (\pm 0.0)$	$1.0 (\pm 0.0)$	$0.844 (\pm 0.125)$	$0.927 (\pm 0.065)$	$0.927 (\pm 0.065)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.847 (\pm 0.032)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.896 (\pm 0.0)$	$0.847 (\pm 0.067)$	$0.847 (\pm 0.032)$	$0.847 (\pm 0.032)$	$0.896 (\pm 0.0)$	$0.91 (\pm 0.024)$
$7 \to 13$	$0.882 (\pm 0.043)$	$0.903 (\pm 0.012)$	$0.903 (\pm 0.012)$	$0.917 (\pm 0.021)$	$0.924 (\pm 0.024)$	$0.882 (\pm 0.043)$	$0.882 (\pm 0.043)$	$0.931 (\pm 0.024)$	$0.944 (\pm 0.012)$
$9 \to 18$	$0.521 (\pm 0.072)$	$0.569 (\pm 0.139)$	$0.569 (\pm 0.139)$	$0.5 (\pm 0.104)$	$0.5 (\pm 0.095)$	$0.535 (\pm 0.064)$	$0.507 (\pm 0.024)$	$0.521 (\pm 0.091)$	$0.688 (\pm 0.075)$
平均值	$0.785 (\pm 0.032)$	$0.812 (\pm 0.033)$	$0.812 (\pm 0.033)$	$0.8 (\pm 0.025)$	$0.759 (\pm 0.065)$	$0.773 (\pm 0.043)$	$0.767 (\pm 0.035)$	$0.807 (\pm 0.023)$	$0.86 (\pm 0.031)$

Task	CoDATS
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.681 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.688 (\pm 0.0)$	$0.694 (\pm 0.024)$	$0.681 (\pm 0.024)$	$0.694 (\pm 0.024)$	$0.694 (\pm 0.012)$	$0.708 (\pm 0.021)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.865 (\pm 0.11)$	$0.865 (\pm 0.1)$	$0.865 (\pm 0.1)$	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.84 (\pm 0.032)$	$0.889 (\pm 0.032)$	$0.889 (\pm 0.032)$	$0.889 (\pm 0.032)$	$0.882 (\pm 0.064)$	$0.84 (\pm 0.048)$	$0.896 (\pm 0.083)$	$0.924 (\pm 0.012)$	$0.951 (\pm 0.043)$
$7 \to 13$	$0.75 (\pm 0.217)$	0.944(±0.012	$0.944 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.917 (\pm 0.036)$	$0.854 (\pm 0.036$	$0.75 (\pm 0.217)$	$0.938 (\pm 0.0)$	$0.951 (\pm 0.012)$
$9 \to 18$	$0.528 (\pm 0.032)$	$0.625 (\pm 0.062)$	$0.625 (\pm 0.062)$	$0.639 (\pm 0.043)$	$0.569 (\pm 0.305)$	$0.764 (\pm 0.12)$	$0.764 (\pm 0.12)$	$0.674 (\pm 0.115)$	$0.826 (\pm 0.012)$
Avg.	$0.76 (\pm 0.061)$	$0.832 (\pm 0.026)$	$0.832 (\pm 0.026)$	$0.832 (\pm 0.017)$	$0.785 (\pm 0.108)$	$0.801 (\pm 0.066)$	$0.794 (\pm 0.109)$	$0.846 (\pm 0.028)$	$0.888 (\pm 0.018)$

任务	CoDATS
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA（我们的）	TB
$12 \to 16$	$0.681 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.688 (\pm 0.0)$	$0.694 (\pm 0.024)$	$0.681 (\pm 0.024)$	$0.694 (\pm 0.024)$	$0.694 (\pm 0.012)$	$0.708 (\pm 0.021)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.865 (\pm 0.11)$	$0.865 (\pm 0.1)$	$0.865 (\pm 0.1)$	1.0(±0.0)	$1.0 (\pm 0.0)$
$6 \to 23$	$0.84 (\pm 0.032)$	$0.889 (\pm 0.032)$	$0.889 (\pm 0.032)$	$0.889 (\pm 0.032)$	$0.882 (\pm 0.064)$	$0.84 (\pm 0.048)$	$0.896 (\pm 0.083)$	$0.924 (\pm 0.012)$	$0.951 (\pm 0.043)$
$7 \to 13$	$0.75 (\pm 0.217)$	0.944(±0.012	$0.944 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.917 (\pm 0.036)$	$0.854 (\pm 0.036$	$0.75 (\pm 0.217)$	$0.938 (\pm 0.0)$	$0.951 (\pm 0.012)$
$9 \to 18$	$0.528 (\pm 0.032)$	$0.625 (\pm 0.062)$	$0.625 (\pm 0.062)$	$0.639 (\pm 0.043)$	$0.569 (\pm 0.305)$	$0.764 (\pm 0.12)$	$0.764 (\pm 0.12)$	$0.674 (\pm 0.115)$	$0.826 (\pm 0.012)$
平均值	$0.76 (\pm 0.061)$	$0.832 (\pm 0.026)$	$0.832 (\pm 0.026)$	$0.832 (\pm 0.017)$	$0.785 (\pm 0.108)$	$0.801 (\pm 0.066)$	$0.794 (\pm 0.109)$	$0.846 (\pm 0.028)$	$0.888 (\pm 0.018)$

Task	Deep-Coral
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.639 (\pm 0.084)$	$0.688 (\pm 0.0)$	$0.688 (\pm 0.0)$
$2 \to 11$	$0.969 (\pm 0.054)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.875 (\pm 0.094)$	$0.875 (\pm 0.083)$	$0.885 (\pm 0.095)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.826 (\pm 0.084)$	$0.882 (\pm 0.012)$	$0.882 (\pm 0.012)$	$0.889 (\pm 0.012)$	$0.896 (\pm 0.0)$	$0.826 (\pm 0.084)$	$0.833 (\pm 0.091)$	$0.896 (\pm 0.0)$	$0.903 (\pm 0.043)$
$7 \to 13$	$0.917 (\pm 0.036)$	0.91(±0.032	$0.91 (\pm 0.032)$	$0.91 (\pm 0.032)$	$0.896 (\pm 0.021)$	$0.917 (\pm 0.036)$	$0.917 (\pm 0.036)$	$0.903 (\pm 0.024)$	$0.931 (\pm 0.012)$
$9 \to 18$	$0.556 (\pm 0.087)$	$0.528 (\pm 0.043)$	$0.528 (\pm 0.043)$	$0.569 (\pm 0.115)$	$0.507 (\pm 0.272)$	$0.542 (\pm 0.144)$	$0.59 (\pm 0.032)$	$0.556 (\pm 0.064)$	$0.59 (\pm 0.221)$
Avg.	$0.79 (\pm 0.055)$	0.8(±0.02)	0.8(±0.02)	$0.808 (\pm 0.034)$	$0.771 (\pm 0.08)$	$0.768 (\pm 0.072)$	$0.773 (\pm 0.068)$	$0.808 (\pm 0.018)$	$0.822 (\pm 0.055)$

任务	深度珊瑚
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	TWV	DEV	IWA (我们的)	TB
$12 \to 16$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.674 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.639 (\pm 0.084)$	$0.688 (\pm 0.0)$	$0.688 (\pm 0.0)$
$2 \to 11$	$0.969 (\pm 0.054)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.875 (\pm 0.094)$	$0.875 (\pm 0.083)$	$0.885 (\pm 0.095)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.826 (\pm 0.084)$	$0.882 (\pm 0.012)$	$0.882 (\pm 0.012)$	$0.889 (\pm 0.012)$	$0.896 (\pm 0.0)$	$0.826 (\pm 0.084)$	$0.833 (\pm 0.091)$	$0.896 (\pm 0.0)$	$0.903 (\pm 0.043)$
$7 \to 13$	$0.917 (\pm 0.036)$	0.91(±0.032)	$0.91 (\pm 0.032)$	$0.91 (\pm 0.032)$	$0.896 (\pm 0.021)$	$0.917 (\pm 0.036)$	$0.917 (\pm 0.036)$	$0.903 (\pm 0.024)$	$0.931 (\pm 0.012)$
$9 \to 18$	$0.556 (\pm 0.087)$	$0.528 (\pm 0.043)$	$0.528 (\pm 0.043)$	$0.569 (\pm 0.115)$	$0.507 (\pm 0.272)$	$0.542 (\pm 0.144)$	$0.59 (\pm 0.032)$	$0.556 (\pm 0.064)$	$0.59 (\pm 0.221)$
平均值	$0.79 (\pm 0.055)$	0.8(±0.02)	0.8(±0.02)	$0.808 (\pm 0.034)$	$0.771 (\pm 0.08)$	$0.768 (\pm 0.072)$	$0.773 (\pm 0.068)$	$0.808 (\pm 0.018)$	$0.822 (\pm 0.055)$

Task	CDAN
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.674 (\pm 0.043)$	$0.722 (\pm 0.012)$	$0.722 (\pm 0.012)$	$0.722 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.674 (\pm 0.043)$	$0.646 (\pm 0.055)$	$0.715 (\pm 0.024)$	$0.729 (\pm 0.036)$
$2 \to 11$	$0.99 (\pm 0.018)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.865 (\pm 0.065)$	$0.792 (\pm 0.28)$	$0.802 (\pm 0.29)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.854 (\pm 0.042)$	$0.924 (\pm 0.024)$	$0.924 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.847 (\pm 0.032)$	$0.708 (\pm 0.295)$	$0.924 (\pm 0.024)$	$0.924 (\pm 0.024)$
$7 \to 13$	$0.826 (\pm 0.139)$	0.951(±0.012)	$0.951 (\pm 0.012)$	$0.951 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.896 (\pm 0.021)$	$0.812 (\pm 0.108)$	$0.951 (\pm 0.012)$	$0.958 (\pm 0.0)$
$9 \to 18$	$0.438 (\pm 0.055)$	$0.618 (\pm 0.064)$	$0.618 (\pm 0.064)$	$0.618 (\pm 0.064)$	$0.597 (\pm 0.098)$	$0.694 (\pm 0.126)$	$0.465 (\pm 0.087)$	$0.639 (\pm 0.048)$	$0.729 (\pm 0.083)$
Avg.	$0.756 (\pm 0.059)$	$0.843 (\pm 0.022)$	$0.843 (\pm 0.022)$	$0.84 (\pm 0.022)$	$0.802 (\pm 0.042)$	$0.781 (\pm 0.1)$	$0.687 (\pm 0.167)$	$0.846 (\pm 0.022)$	$0.868 (\pm 0.029)$

任务	CDAN
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$12 \to 16$	$0.674 (\pm 0.043)$	$0.722 (\pm 0.012)$	$0.722 (\pm 0.012)$	$0.722 (\pm 0.012)$	$0.694 (\pm 0.012)$	$0.674 (\pm 0.043)$	$0.646 (\pm 0.055)$	$0.715 (\pm 0.024)$	$0.729 (\pm 0.036)$
$2 \to 11$	$0.99 (\pm 0.018)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.865 (\pm 0.065)$	$0.792 (\pm 0.28)$	$0.802 (\pm 0.29)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.854 (\pm 0.042)$	$0.924 (\pm 0.024)$	$0.924 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.91 (\pm 0.024)$	$0.847 (\pm 0.032)$	$0.708 (\pm 0.295)$	$0.924 (\pm 0.024)$	$0.924 (\pm 0.024)$
$7 \to 13$	$0.826 (\pm 0.139)$	0.951(±0.012)	$0.951 (\pm 0.012)$	$0.951 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.896 (\pm 0.021)$	$0.812 (\pm 0.108)$	$0.951 (\pm 0.012)$	$0.958 (\pm 0.0)$
$9 \to 18$	$0.438 (\pm 0.055)$	$0.618 (\pm 0.064)$	$0.618 (\pm 0.064)$	$0.618 (\pm 0.064)$	$0.597 (\pm 0.098)$	$0.694 (\pm 0.126)$	$0.465 (\pm 0.087)$	$0.639 (\pm 0.048)$	$0.729 (\pm 0.083)$
平均	$0.756 (\pm 0.059)$	$0.843 (\pm 0.022)$	$0.843 (\pm 0.022)$	$0.84 (\pm 0.022)$	$0.802 (\pm 0.042)$	$0.781 (\pm 0.1)$	$0.687 (\pm 0.167)$	$0.846 (\pm 0.022)$	$0.868 (\pm 0.029)$

Task	DANN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.688 (\pm 0.021)$	$0.701 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.715 (\pm 0.012)$	$0.701 (\pm 0.012)$	$0.701 (\pm 0.012)$	$0.729 (\pm 0.0)$	$0.722 (\pm 0.0)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.729 (\pm 0.141)$	$0.76 (\pm 0.148)$	$0.802 (\pm 0.203)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.882 (\pm 0.024)$	$0.917 (\pm 0.021)$	$0.917 (\pm 0.021)$	$0.931 (\pm 0.012)$	$0.889 (\pm 0.032)$	$0.868 (\pm 0.024)$	$0.868 (\pm 0.024)$	$0.931 (\pm 0.012)$	$0.958 (\pm 0.036)$
$7 \to 13$	$0.806 (\pm 0.127)$	$0.944 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.938 (\pm 0.021)$	$0.903 (\pm 0.024)$	$0.944 (\pm 0.048$	$0.944 (\pm 0.012)$	$0.965 (\pm 0.012)$
$9 \to 18$	$0.403 (\pm 0.12)$	$0.715 (\pm 0.012)$	$0.715 (\pm 0.012)$	$0.701 (\pm 0.067)$	$0.729 (\pm 0.127)$	$0.583 (\pm 0.11)$	$0.583 (\pm 0.11)$	$0.639 (\pm 0.103)$	$0.674 (\pm 0.127)$
Avg.	$0.756 (\pm 0.058)$	$0.856 (\pm 0.014)$	$0.856 (\pm 0.014)$	$0.856 (\pm 0.023)$	$0.8 (\pm 0.066)$	$0.763 (\pm 0.064)$	$0.78 (\pm 0.08)$	$0.849 (\pm 0.025)$	$0.864 (\pm 0.035)$

任务	DANN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$12 \to 16$	$0.688 (\pm 0.021)$	$0.701 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.701 (\pm 0.024)$	$0.715 (\pm 0.012)$	$0.701 (\pm 0.012)$	$0.701 (\pm 0.012)$	$0.729 (\pm 0.0)$	$0.722 (\pm 0.0)$
$2 \to 11$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.729 (\pm 0.141)$	$0.76 (\pm 0.148)$	$0.802 (\pm 0.203)$	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.882 (\pm 0.024)$	$0.917 (\pm 0.021)$	$0.917 (\pm 0.021)$	$0.931 (\pm 0.012)$	$0.889 (\pm 0.032)$	$0.868 (\pm 0.024)$	$0.868 (\pm 0.024)$	$0.931 (\pm 0.012)$	$0.958 (\pm 0.036)$
$7 \to 13$	$0.806 (\pm 0.127)$	$0.944 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.944 (\pm 0.012)$	$0.938 (\pm 0.021)$	$0.903 (\pm 0.024)$	$0.944 (\pm 0.048$	$0.944 (\pm 0.012)$	$0.965 (\pm 0.012)$
$9 \to 18$	$0.403 (\pm 0.12)$	$0.715 (\pm 0.012)$	$0.715 (\pm 0.012)$	$0.701 (\pm 0.067)$	$0.729 (\pm 0.127)$	$0.583 (\pm 0.11)$	$0.583 (\pm 0.11)$	$0.639 (\pm 0.103)$	$0.674 (\pm 0.127)$
平均值	$0.756 (\pm 0.058)$	$0.856 (\pm 0.014)$	$0.856 (\pm 0.014)$	$0.856 (\pm 0.023)$	$0.8 (\pm 0.066)$	$0.763 (\pm 0.064)$	$0.78 (\pm 0.08)$	$0.849 (\pm 0.025)$	$0.864 (\pm 0.035)$

Task	DSAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$12 \to 16$	$0.681 (\pm 0.012)$	$0.688 (\pm 0.021)$	$0.688 (\pm 0.021)$	$0.715 (\pm 0.024)$	$0.694 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.556 (\pm 0.229)$	$0.715 (\pm 0.024)$	$0.736 (\pm 0.012)$
$2 \to 11$	$1.0 (\pm 0.0)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.969 (\pm 0.0)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.764 (\pm 0.094)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.917 (\pm 0.021)$	$0.847 (\pm 0.052)$	$0.785 (\pm 0.103)$	$0.785 (\pm 0.103)$	$0.931 (\pm 0.012)$	$0.938 (\pm 0.0)$
$7 \to 13$	$0.854 (\pm 0.036)$	$0.951 (\pm 0.012)$	$0.951 (\pm 0.012)$	$0.951 (\pm 0.012)$	$0.854 (\pm 0.036$	$0.854 (\pm 0.036)$	$0.854 (\pm 0.036)$	$0.951 (\pm 0.012)$	$0.958 (\pm 0.0)$
$9 \to 18$	$0.514 (\pm 0.032)$	$0.639 (\pm 0.079)$	$0.639 (\pm 0.079)$	$0.688 (\pm 0.091$	$0.382 (\pm 0.012)$	$0.556 (\pm 0.067)$	$0.528 (\pm 0.067)$	$0.694 (\pm 0.032)$	$0.743 (\pm 0.087)$
Avg.	$0.762 (\pm 0.035)$	$0.843 (\pm 0.022)$	$0.843 (\pm 0.022)$	$0.854 (\pm 0.03)$	$0.749 (\pm 0.023)$	$0.775 (\pm 0.044)$	$0.744 (\pm 0.087)$	$0.858 (\pm 0.016)$	$0.875 (\pm 0.02)$

任务	DSAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$12 \to 16$	$0.681 (\pm 0.012)$	$0.688 (\pm 0.021)$	$0.688 (\pm 0.021)$	$0.715 (\pm 0.024)$	$0.694 (\pm 0.012)$	$0.681 (\pm 0.012)$	$0.556 (\pm 0.229)$	$0.715 (\pm 0.024)$	$0.736 (\pm 0.012)$
$2 \to 11$	$1.0 (\pm 0.0)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	$0.969 (\pm 0.0)$	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)	1.0(±0.0)
$6 \to 23$	$0.764 (\pm 0.094)$	$0.938 (\pm 0.0)$	$0.938 (\pm 0.0)$	$0.917 (\pm 0.021)$	$0.847 (\pm 0.052)$	$0.785 (\pm 0.103)$	$0.785 (\pm 0.103)$	$0.931 (\pm 0.012)$	$0.938 (\pm 0.0)$
$7 \to 13$	$0.854 (\pm 0.036)$	$0.951 (\pm 0.012)$	$0.951 (\pm 0.012)$	$0.951 (\pm 0.012)$	$0.854 (\pm 0.036$	$0.854 (\pm 0.036)$	$0.854 (\pm 0.036)$	$0.951 (\pm 0.012)$	$0.958 (\pm 0.0)$
$9 \to 18$	$0.514 (\pm 0.032)$	$0.639 (\pm 0.079)$	$0.639 (\pm 0.079)$	$0.688 (\pm 0.091$	$0.382 (\pm 0.012)$	$0.556 (\pm 0.067)$	$0.528 (\pm 0.067)$	$0.694 (\pm 0.032)$	$0.743 (\pm 0.087)$
平均值	$0.762 (\pm 0.035)$	$0.843 (\pm 0.022)$	$0.843 (\pm 0.022)$	$0.854 (\pm 0.03)$	$0.749 (\pm 0.023)$	$0.775 (\pm 0.044)$	$0.744 (\pm 0.087)$	$0.858 (\pm 0.016)$	$0.875 (\pm 0.02)$

Task	HoMM
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.719 (\pm 0.041)$	$0.737 (\pm 0.019)$	$0.737 (\pm 0.019)$	$0.735 (\pm 0.024)$	$0.635 (\pm 0.122)$	$0.704 (\pm 0.011)$	$0.703 (\pm 0.013)$	$0.732 (\pm 0.01)$	$0.733 (\pm 0.019)$
$1 \to 6$	$0.788 (\pm 0.11)$	$0.871 (\pm 0.011)$	$0.871 (\pm 0.011)$	$0.869 (\pm 0.013)$	$0.788 (\pm 0.163)$	$0.749 (\pm 0.06)$	$0.785 (\pm 0.106)$	$0.879 (\pm 0.011)$	$0.914 (\pm 0.01)$
$2 \to 7$	$0.528 (\pm 0.103)$	$0.457 (\pm 0.007)$	$0.457 (\pm 0.007)$	$0.461 (\pm 0.007)$	0.469(±0.051	$0.497 (\pm 0.105)$	$0.516 (\pm 0.087)$	$0.455 (\pm 0.015)$	$0.546 (\pm 0.071)$
$3 \to 8$	$0.797 (\pm 0.007)$	0.818(±0.002	$0.818 (\pm 0.002)$	$0.816 (\pm {0.007}^{\circ})$	$0.812 (\pm 0.014)$	$0.805 (\pm 0.01)$	$0.805 (\pm 0.01)$	$0.818 (\pm 0.005)$	$0.831 \dot{i} \pm {0.022}^{\dot{}}$
$4 \to 5$	$0.861 \dot{1} \pm 0.022 \dot{2}$	$0.914 (\pm 0.007)$	0.914(±0.007)	0.911/±0.005	$0.798 (\pm 0.164)$	$0.844 (\pm 0.016)$	$0.855 (\pm 0.01)$	0.911(±0.005	$0.94 (\pm 0.04)$
Avg.	$0.739 (\pm 0.057)$	$0.759 (\pm 0.009)$	$0.759 (\pm 0.009)$	$0.759 (\pm 0.011)$	$0.7 (\pm 0.103)$	$0.72 (\pm 0.041)$	$0.733 (\pm 0.045)$	$0.759 (\pm 0.009)$	$0.793 (\pm 0.032)$

任务	HoMM
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.719 (\pm 0.041)$	$0.737 (\pm 0.019)$	$0.737 (\pm 0.019)$	$0.735 (\pm 0.024)$	$0.635 (\pm 0.122)$	$0.704 (\pm 0.011)$	$0.703 (\pm 0.013)$	$0.732 (\pm 0.01)$	$0.733 (\pm 0.019)$
$1 \to 6$	$0.788 (\pm 0.11)$	$0.871 (\pm 0.011)$	$0.871 (\pm 0.011)$	$0.869 (\pm 0.013)$	$0.788 (\pm 0.163)$	$0.749 (\pm 0.06)$	$0.785 (\pm 0.106)$	$0.879 (\pm 0.011)$	$0.914 (\pm 0.01)$
$2 \to 7$	$0.528 (\pm 0.103)$	$0.457 (\pm 0.007)$	$0.457 (\pm 0.007)$	$0.461 (\pm 0.007)$	0.469(±0.051	$0.497 (\pm 0.105)$	$0.516 (\pm 0.087)$	$0.455 (\pm 0.015)$	$0.546 (\pm 0.071)$
$3 \to 8$	$0.797 (\pm 0.007)$	0.818(±0.002	$0.818 (\pm 0.002)$	$0.816 (\pm {0.007}^{\circ})$	$0.812 (\pm 0.014)$	$0.805 (\pm 0.01)$	$0.805 (\pm 0.01)$	$0.818 (\pm 0.005)$	$0.831 \dot{i} \pm {0.022}^{\dot{}}$
$4 \to 5$	$0.861 \dot{1} \pm 0.022 \dot{2}$	$0.914 (\pm 0.007)$	0.914(±0.007)	0.911/±0.005	$0.798 (\pm 0.164)$	$0.844 (\pm 0.016)$	$0.855 (\pm 0.01)$	0.911(±0.005	$0.94 (\pm 0.04)$
平均值	$0.739 (\pm 0.057)$	$0.759 (\pm 0.009)$	$0.759 (\pm 0.009)$	$0.759 (\pm 0.011)$	$0.7 (\pm 0.103)$	$0.72 (\pm 0.041)$	$0.733 (\pm 0.045)$	$0.759 (\pm 0.009)$	$0.793 (\pm 0.032)$

Task	AdvSKM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.661 (\pm 0.035)$	$0.722 (\pm 0.009)$	$0.722 (\pm 0.009)$	$0.725 (\pm 0.007)$	$0.629 (\pm 0.076)$	$0.699 (\pm 0.013)$	$0.699 (\pm 0.013)$	$0.718 (\pm 0.009)$	$0.731 (\pm 0.013)$
$1 \to 6$	$0.821 (\pm 0.046)$	$0.849 (\pm 0.002)$	$0.849 (\pm 0.002)$	$0.851 (\pm 0.006)$	$0.604 (\pm 0.045)$	$0.806 (\pm 0.026)$	$0.806 (\pm 0.026)$	$0.857 (\pm 0.006$	$0.838 (\pm 0.018)$
$2 \to 7$	$0.455 (\pm 0.07)$	$0.473 (\pm 0.031)$	$0.473 (\pm 0.031)$	$0.484 (\pm 0.03)$	$0.531 (\pm 0.062$	$0.497 (\pm 0.067)$	$0.497 (\pm 0.067$	$0.49 (\pm 0.014)$	$0.574 (\pm 0.112)$
$3 \to 8$	$0.79 (\pm 0.011)$	$0.799 (\pm 0.008)$	$0.799 (\pm 0.008)$	$0.803 (\pm 0.005)$	$0.805 (\pm 0.01)$	$0.799 (\pm 0.005$	$0.788 (\pm 0.023)$	$0.81 (\pm 0.002)$	$0.818 (\pm 0.012)$
$4 \to 5$	$0.862 (\pm 0.055)$	$0.866 (\pm 0.013)$	$0.866 (\pm 0.013)$	$0.875 (\pm 0.008)$	$0.809 (\pm 0.094)$	$0.85 (\pm 0.025)$	$0.85 (\pm 0.025)$	$0.884 (\pm 0.005)$	$0.895 (\pm 0.014)$
Avg.	$0.718 (\pm 0.044)$	$0.742 (\pm 0.013)$	$0.742 (\pm 0.013)$	$0.748 (\pm 0.011)$	$0.676 (\pm 0.058)$	$0.73 (\pm 0.027)$	$0.728 (\pm 0.031)$	$0.752 (\pm 0.007)$	$0.771 (\pm 0.034)$

任务	AdvSKM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.661 (\pm 0.035)$	$0.722 (\pm 0.009)$	$0.722 (\pm 0.009)$	$0.725 (\pm 0.007)$	$0.629 (\pm 0.076)$	$0.699 (\pm 0.013)$	$0.699 (\pm 0.013)$	$0.718 (\pm 0.009)$	$0.731 (\pm 0.013)$
$1 \to 6$	$0.821 (\pm 0.046)$	$0.849 (\pm 0.002)$	$0.849 (\pm 0.002)$	$0.851 (\pm 0.006)$	$0.604 (\pm 0.045)$	$0.806 (\pm 0.026)$	$0.806 (\pm 0.026)$	$0.857 (\pm 0.006$	$0.838 (\pm 0.018)$
$2 \to 7$	$0.455 (\pm 0.07)$	$0.473 (\pm 0.031)$	$0.473 (\pm 0.031)$	$0.484 (\pm 0.03)$	$0.531 (\pm 0.062$	$0.497 (\pm 0.067)$	$0.497 (\pm 0.067$	$0.49 (\pm 0.014)$	$0.574 (\pm 0.112)$
$3 \to 8$	$0.79 (\pm 0.011)$	$0.799 (\pm 0.008)$	$0.799 (\pm 0.008)$	$0.803 (\pm 0.005)$	$0.805 (\pm 0.01)$	$0.799 (\pm 0.005$	$0.788 (\pm 0.023)$	$0.81 (\pm 0.002)$	$0.818 (\pm 0.012)$
$4 \to 5$	$0.862 (\pm 0.055)$	$0.866 (\pm 0.013)$	$0.866 (\pm 0.013)$	$0.875 (\pm 0.008)$	$0.809 (\pm 0.094)$	$0.85 (\pm 0.025)$	$0.85 (\pm 0.025)$	$0.884 (\pm 0.005)$	$0.895 (\pm 0.014)$
平均	$0.718 (\pm 0.044)$	$0.742 (\pm 0.013)$	$0.742 (\pm 0.013)$	$0.748 (\pm 0.011)$	$0.676 (\pm 0.058)$	$0.73 (\pm 0.027)$	$0.728 (\pm 0.031)$	$0.752 (\pm 0.007)$	$0.771 (\pm 0.034)$

Task	DIRT
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	1WV	DEV	IWA (ours)	TB
$0 \to 6$	$0.708 (\pm 0.011)$	$0.571 (\pm 0.079)$	$0.571 (\pm 0.079)$	$0.633 (\pm 0.137)$	$0.629 (\pm 0.085)$	$0.708 (\pm 0.011)$	$0.708 (\pm 0.011)$	$0.779 (\pm 0.011)$	$0.739 (\pm 0.011)$
$1 \to 6$	$0.756 (\pm 0.058)$	$0.938 (\pm 0.004$	$0.938 (\pm 0.004)$	$0.938 (\pm 0.004)$	$0.904 (\pm 0.058)$	$0.814 (\pm 0.056)$	$0.756 (\pm 0.058)$	$0.938 (\pm 0.0)$	$0.942 (\pm 0.004)$
$2 \to 7$	$0.531 (\pm 0.057)$	$0.622 (\pm 0.091)$	$0.622 (\pm 0.091)$	$0.621 (\pm 0.09)$	0.658(±0.03)	$0.548 (\pm 0.049)$	0.548(±0.049	$0.53 (\pm 0.025)$	$0.688 (\pm 0.004)$
$3 \to 8$	$0.807 (\pm 0.008)$	$0.846 (\pm 0.022)$	$0.846 (\pm 0.022)$	$0.837 (\pm 0.015)$	0.857(±0.087)	$0.807 (\pm 0.008)$	$0.807 (\pm 0.008)$	$0.848 (\pm 0.007)$	$0.911 (\pm 0.07)$
$4 \to 5$	$0.839 (\pm 0.016)$	$0.984 (\pm 0.004$	$0.984 (\pm 0.004)$	$0.984 (\pm 0.004)$	$0.93 (\pm 0.061)$	$0.839 (\pm 0.016)$	$0.878 (\pm 0.074)$	$0.984 (\pm 0.004)$	$0.984 (\pm 0.002)$
Avg.	$0.728 (\pm 0.03)$	$0.792 (\pm 0.04)$	$0.792 (\pm 0.04)$	$0.803 (\pm 0.05)$	$0.796 (\pm 0.064)$	$0.743 (\pm 0.028)$	$0.739 (\pm 0.04)$	$0.816 (\pm 0.009)$	$0.853 (\pm 0.018)$

任务	DIRT
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	1WV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.708 (\pm 0.011)$	$0.571 (\pm 0.079)$	$0.571 (\pm 0.079)$	$0.633 (\pm 0.137)$	$0.629 (\pm 0.085)$	$0.708 (\pm 0.011)$	$0.708 (\pm 0.011)$	$0.779 (\pm 0.011)$	$0.739 (\pm 0.011)$
$1 \to 6$	$0.756 (\pm 0.058)$	$0.938 (\pm 0.004$	$0.938 (\pm 0.004)$	$0.938 (\pm 0.004)$	$0.904 (\pm 0.058)$	$0.814 (\pm 0.056)$	$0.756 (\pm 0.058)$	$0.938 (\pm 0.0)$	$0.942 (\pm 0.004)$
$2 \to 7$	$0.531 (\pm 0.057)$	$0.622 (\pm 0.091)$	$0.622 (\pm 0.091)$	$0.621 (\pm 0.09)$	0.658(±0.03)	$0.548 (\pm 0.049)$	0.548(±0.049)	$0.53 (\pm 0.025)$	$0.688 (\pm 0.004)$
$3 \to 8$	$0.807 (\pm 0.008)$	$0.846 (\pm 0.022)$	$0.846 (\pm 0.022)$	$0.837 (\pm 0.015)$	0.857(±0.087)	$0.807 (\pm 0.008)$	$0.807 (\pm 0.008)$	$0.848 (\pm 0.007)$	$0.911 (\pm 0.07)$
$4 \to 5$	$0.839 (\pm 0.016)$	$0.984 (\pm 0.004$	$0.984 (\pm 0.004)$	$0.984 (\pm 0.004)$	$0.93 (\pm 0.061)$	$0.839 (\pm 0.016)$	$0.878 (\pm 0.074)$	$0.984 (\pm 0.004)$	$0.984 (\pm 0.002)$
平均值	$0.728 (\pm 0.03)$	$0.792 (\pm 0.04)$	$0.792 (\pm 0.04)$	$0.803 (\pm 0.05)$	$0.796 (\pm 0.064)$	$0.743 (\pm 0.028)$	$0.739 (\pm 0.04)$	$0.816 (\pm 0.009)$	$0.853 (\pm 0.018)$

Task	DDC
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.575 (\pm 0.083)$	0.693(±0.017)	$0.693 (\pm 0.017)$	$0.672 (\pm 0.009)$	$0.632 (\pm 0.038)$	$0.646 (\pm 0.015)$	$0.6 (\pm 0.084)$	$0.697 (\pm 0.013)$	$0.651 (\pm 0.013)$
$1 \to 6$	$0.875 (\pm 0.014)$	$0.888 (\pm 0.007)$	$0.888 (\pm 0.007)$	$0.894 (\pm 0.013)$	$0.886 (\pm 0.019)$	$0.856 (\pm 0.067)$	$0.856 (\pm 0.067)$	$0.882 (\pm 0.017)$	$0.899 (\pm 0.016)$
$2 \to 7$	$0.487 (\pm 0.016)$	$0.457 (\pm 0.03)$	$0.457 (\pm 0.03)$	$0.455 (\pm 0.029)$	0.497(±0.028	$0.438 (\pm 0.054)$	$0.438 (\pm 0.054)$	$0.439 (\pm 0.041)$	$0.533 (\pm {0.107}^{\circ})$
$3 \to 8$	$0.815 (\pm 0.019)$	$0.805 (\pm 0.014)$	$0.805 (\pm 0.014)$	0.8071 $\pm 0.016$	$0.797 (\pm 0.022)$	$0.827 \dot{(} \pm {0.025}^{\dot{}}$	$0.831 (\pm 0.02)$	$0.818 (\pm 0.008)$	$0.822 (\pm 0.02)$
$4 \to 5$	$0.831 (\pm 0.048)$	$0.908 (\pm 0.011)$	$0.908 (\pm 0.011)$	$0.911 (\pm 0.01)$	$0.772 (\pm 0.085)$	$0.792 (\pm 0.02)$	$0.801 (\pm 0.012)$	$0.905 (\pm 0.008)$	$0.888 (\pm 0.01)$
Avg.	$0.716 (\pm 0.036)$	$0.75 (\pm 0.016)$	$0.75 (\pm 0.016)$	$0.748 (\pm 0.015)$	$0.717 (\pm 0.038)$	$0.711 (\pm 0.036)$	$0.705 (\pm 0.047)$	$0.748 (\pm 0.017)$	$0.758 (\pm 0.033)$

任务	DDC
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.575 (\pm 0.083)$	0.693(±0.017)	$0.693 (\pm 0.017)$	$0.672 (\pm 0.009)$	$0.632 (\pm 0.038)$	$0.646 (\pm 0.015)$	$0.6 (\pm 0.084)$	$0.697 (\pm 0.013)$	$0.651 (\pm 0.013)$
$1 \to 6$	$0.875 (\pm 0.014)$	$0.888 (\pm 0.007)$	$0.888 (\pm 0.007)$	$0.894 (\pm 0.013)$	$0.886 (\pm 0.019)$	$0.856 (\pm 0.067)$	$0.856 (\pm 0.067)$	$0.882 (\pm 0.017)$	$0.899 (\pm 0.016)$
$2 \to 7$	$0.487 (\pm 0.016)$	$0.457 (\pm 0.03)$	$0.457 (\pm 0.03)$	$0.455 (\pm 0.029)$	0.497(±0.028)	$0.438 (\pm 0.054)$	$0.438 (\pm 0.054)$	$0.439 (\pm 0.041)$	$0.533 (\pm {0.107}^{\circ})$
$3 \to 8$	$0.815 (\pm 0.019)$	$0.805 (\pm 0.014)$	$0.805 (\pm 0.014)$	0.8071 $\pm 0.016$	$0.797 (\pm 0.022)$	$0.827 \dot{(} \pm {0.025}^{\dot{}}$	$0.831 (\pm 0.02)$	$0.818 (\pm 0.008)$	$0.822 (\pm 0.02)$
$4 \to 5$	$0.831 (\pm 0.048)$	$0.908 (\pm 0.011)$	$0.908 (\pm 0.011)$	$0.911 (\pm 0.01)$	$0.772 (\pm 0.085)$	$0.792 (\pm 0.02)$	$0.801 (\pm 0.012)$	$0.905 (\pm 0.008)$	$0.888 (\pm 0.01)$
平均值	$0.716 (\pm 0.036)$	$0.75 (\pm 0.016)$	$0.75 (\pm 0.016)$	$0.748 (\pm 0.015)$	$0.717 (\pm 0.038)$	$0.711 (\pm 0.036)$	$0.705 (\pm 0.047)$	$0.748 (\pm 0.017)$	$0.758 (\pm 0.033)$

Task	CMD
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.703 (\pm 0.04)$	$0.643 (\pm 0.046)$	$0.643 (\pm 0.046)$	$0.679 (\pm 0.054)$	$0.672 (\pm 0.103)$	$0.743 (\pm 0.021)$	$0.743 (\pm 0.021)$	$0.693 (\pm 0.012)$	$0.724 (\pm 0.021)$
$1 \to 6$	$0.861 (\pm 0.035)$	0.915(±0.009	$0.915 (\pm 0.009)$	$0.912 (\pm 0.0)$	$0.826 (\pm 0.103)$	$0.899 (\pm 0.03)$	$0.843 (\pm 0.156)$	$0.907 (\pm 0.006)$	$0.925 (\pm 0.004)$
$2 \to 7$	$0.573 (\pm 0.018)$	$0.494 (\pm 0.011)$	$0.494 (\pm 0.011)$	$0.499 (\pm 0.023)$	$0.51 (\pm 0.066)$	$0.577 (\pm 0.011)$	$0.557 (\pm 0.039)$	$0.482 (\pm 0.008)$	$0.603 (\pm 0.016)$
$3 \to 8$	$0.799 (\pm 0.016)$	$0.816 (\pm 0.0)$	$0.816 (\pm 0.0)$	$0.816 (\pm 0.0)$	$0.797 (\pm 0.004)$	$0.799 (\pm 0.016)$	$0.738 (\pm 0.109)$	$0.811 (\pm 0.011)$	$0.822 (\pm 0.01)$
$4 \to 5$	$0.806 (\pm 0.024)$	$0.952 (\pm 0.016)$	$0.952 (\pm 0.016)$	$0.93 (\pm 0.007)$	$0.879 (\pm 0.054)$	$0.854 (\pm 0.063)$	$0.332 (\pm 0.051)$	$0.939 (\pm 0.002)$	$0.961 (\pm 0.024)$
Avg.	$0.748 (\pm 0.026)$	$0.764 (\pm 0.016)$	$0.764 (\pm 0.016)$	$0.767 (\pm 0.017)$	$0.737 (\pm 0.066)$	$0.775 (\pm 0.028)$	$0.643 (\pm 0.075)$	$0.766 (\pm 0.008)$	$0.807 (\pm 0.015)$

任务	CMD
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.703 (\pm 0.04)$	$0.643 (\pm 0.046)$	$0.643 (\pm 0.046)$	$0.679 (\pm 0.054)$	$0.672 (\pm 0.103)$	$0.743 (\pm 0.021)$	$0.743 (\pm 0.021)$	$0.693 (\pm 0.012)$	$0.724 (\pm 0.021)$
$1 \to 6$	$0.861 (\pm 0.035)$	0.915(±0.009	$0.915 (\pm 0.009)$	$0.912 (\pm 0.0)$	$0.826 (\pm 0.103)$	$0.899 (\pm 0.03)$	$0.843 (\pm 0.156)$	$0.907 (\pm 0.006)$	$0.925 (\pm 0.004)$
$2 \to 7$	$0.573 (\pm 0.018)$	$0.494 (\pm 0.011)$	$0.494 (\pm 0.011)$	$0.499 (\pm 0.023)$	$0.51 (\pm 0.066)$	$0.577 (\pm 0.011)$	$0.557 (\pm 0.039)$	$0.482 (\pm 0.008)$	$0.603 (\pm 0.016)$
$3 \to 8$	$0.799 (\pm 0.016)$	$0.816 (\pm 0.0)$	$0.816 (\pm 0.0)$	$0.816 (\pm 0.0)$	$0.797 (\pm 0.004)$	$0.799 (\pm 0.016)$	$0.738 (\pm 0.109)$	$0.811 (\pm 0.011)$	$0.822 (\pm 0.01)$
$4 \to 5$	$0.806 (\pm 0.024)$	$0.952 (\pm 0.016)$	$0.952 (\pm 0.016)$	$0.93 (\pm 0.007)$	$0.879 (\pm 0.054)$	$0.854 (\pm 0.063)$	$0.332 (\pm 0.051)$	$0.939 (\pm 0.002)$	$0.961 (\pm 0.024)$
平均	$0.748 (\pm 0.026)$	$0.764 (\pm 0.016)$	$0.764 (\pm 0.016)$	$0.767 (\pm 0.017)$	$0.737 (\pm 0.066)$	$0.775 (\pm 0.028)$	$0.643 (\pm 0.075)$	$0.766 (\pm 0.008)$	$0.807 (\pm 0.015)$

Task	MMDA
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.706 (\pm 0.042)$	0.737(±0.021)	$0.737 (\pm 0.021)$	$0.737 (\pm 0.018)$	$0.704 (\pm 0.037)$	$0.699 (\pm 0.055)$	$0.725 (\pm 0.018)$	$0.746 (\pm 0.008)$	$0.732 (\pm 0.008)$
$1 \to 6$	$0.792 (\pm 0.098)$	$0.91 (\pm 0.002)$	$0.91 (\pm 0.002)$	$0.91 (\pm 0.002)$	$0.742 (\pm 0.147)$	$0.732 (\pm 0.009)$	$0.722 (\pm 0.009)$	$0.897 (\pm 0.002)$	$0.914 (\pm 0.028)$
$2 \to 7$	$0.552 (\pm 0.073)$	$0.481 (\pm 0.003)$	$0.481 (\pm 0.003)$	$0.482 (\pm 0.004)$	$0.515 (\pm 0.058$	$0.496 (\pm 0.027)$	$0.543 (\pm 0.055$	$0.488 (\pm 0.01)$	$0.552 (\pm 0.073)$
$3 \to 8$	$0.79 (\pm 0.016)$	0.862(±0.026	$0.862 (\pm 0.026)$	$0.841 (\pm 0.01)$	$0.794 (\pm 0.022)$	$0.802 (\pm 0.018)$	$0.802 (\pm 0.018)$	$0.839 (\pm 0.012)$	$0.932 (\pm 0.005)$
$4 \to 5$	$0.852 (\pm 0.036)$	$0.917 (\pm 0.016)$	$0.917 (\pm 0.016)$	$0.931 (\pm 0.002)$	$0.733 (\pm 0.061)$	$0.865 (\pm 0.055)$	$0.865 (\pm 0.055)$	$0.928 (\pm 0.006$	$0.947 (\pm 0.022)$
Avg.	$0.738 (\pm 0.053)$	$0.781 (\pm 0.014)$	$0.781 (\pm 0.014)$	$0.78 (\pm 0.007)$	$0.698 (\pm 0.065)$	$0.719 (\pm 0.033)$	$0.731 (\pm 0.031)$	$0.78 (\pm 0.008)$	$0.815 (\pm 0.027)$

任务	MMDA
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$0 \to 6$	$0.706 (\pm 0.042)$	0.737(±0.021)	$0.737 (\pm 0.021)$	$0.737 (\pm 0.018)$	$0.704 (\pm 0.037)$	$0.699 (\pm 0.055)$	$0.725 (\pm 0.018)$	$0.746 (\pm 0.008)$	$0.732 (\pm 0.008)$
$1 \to 6$	$0.792 (\pm 0.098)$	$0.91 (\pm 0.002)$	$0.91 (\pm 0.002)$	$0.91 (\pm 0.002)$	$0.742 (\pm 0.147)$	$0.732 (\pm 0.009)$	$0.722 (\pm 0.009)$	$0.897 (\pm 0.002)$	$0.914 (\pm 0.028)$
$2 \to 7$	$0.552 (\pm 0.073)$	$0.481 (\pm 0.003)$	$0.481 (\pm 0.003)$	$0.482 (\pm 0.004)$	$0.515 (\pm 0.058$	$0.496 (\pm 0.027)$	$0.543 (\pm 0.055$	$0.488 (\pm 0.01)$	$0.552 (\pm 0.073)$
$3 \to 8$	$0.79 (\pm 0.016)$	0.862(±0.026)	$0.862 (\pm 0.026)$	$0.841 (\pm 0.01)$	$0.794 (\pm 0.022)$	$0.802 (\pm 0.018)$	$0.802 (\pm 0.018)$	$0.839 (\pm 0.012)$	$0.932 (\pm 0.005)$
$4 \to 5$	$0.852 (\pm 0.036)$	$0.917 (\pm 0.016)$	$0.917 (\pm 0.016)$	$0.931 (\pm 0.002)$	$0.733 (\pm 0.061)$	$0.865 (\pm 0.055)$	$0.865 (\pm 0.055)$	$0.928 (\pm 0.006$	$0.947 (\pm 0.022)$
平均	$0.738 (\pm 0.053)$	$0.781 (\pm 0.014)$	$0.781 (\pm 0.014)$	$0.78 (\pm 0.007)$	$0.698 (\pm 0.065)$	$0.719 (\pm 0.033)$	$0.731 (\pm 0.031)$	$0.78 (\pm 0.008)$	$0.815 (\pm 0.027)$

Task	CoDATS
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.64 (\pm 0.096)$	$0.497 (\pm 0.005)$	$0.497 (\pm 0.005)$	$0.499 (\pm 0.006)$	$0.582 (\pm 0.122)$	$0.689 (\pm 0.069)$	$0.606 (\pm 0.106)$	$0.718 (\pm 0.017)$	$0.735 (\pm 0.01)$
$1 \to 6$	$0.775 (\pm 0.091)$	$0.946 (\pm 0.0)$	$0.946 (\pm 0.0)$	$0.946 (\pm 0.0)$	$0.683 (\pm 0.076)$	$0.896 (\pm 0.069)$	$0.896 (\pm 0.069)$	$0.939 (\pm 0.006)$	0.947(±0.009)
$2 \to 7$	$0.527 (\pm 0.076)$	$0.473 (\pm 0.012)$	$0.473 (\pm 0.012)$	$0.475 (\pm 0.01)$	$0.534 (\pm 0.052)$	$0.472 (\pm 0.019)$	$0.555 (\pm 0.069)$	$0.472 (\pm 0.005)$	$0.558 (\pm 0.041)$
$3 \to 8$	$0.783 (\pm 0.015)$	0.971(±0.018	$0.971 (\pm 0.018)$	$0.969 (\pm 0.018)$	$0.97 (\pm 0.025)$	$0.789 (\pm 0.01)$	$0.789 (\pm 0.01)$	$0.96 (\pm 0.03)$	$0.987 (\pm 0.002)$
$4 \to 5$	$0.827 (\pm 0.039)$	$0.971 (\pm 0.013)$	$0.971 (\pm 0.013)$	$0.978 (\pm 0.002)$	$0.841 (\pm 0.1)$	$0.849 (\pm 0.074)$	$0.849 (\pm 0.074)$	$0.973 (\pm 0.004)$	$0.979 (\pm 0.006)$
Avg.	$0.71 (\pm 0.063)$	$0.772 (\pm 0.009)$	$0.772 (\pm 0.009)$	$0.778 (\pm 0.007)$	$0.722 (\pm 0.075)$	$0.739 (\pm 0.048)$	$0.739 (\pm 0.066)$	$0.812 (\pm 0.012)$	$0.841 (\pm 0.014)$

任务	CoDATS
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.64 (\pm 0.096)$	$0.497 (\pm 0.005)$	$0.497 (\pm 0.005)$	$0.499 (\pm 0.006)$	$0.582 (\pm 0.122)$	$0.689 (\pm 0.069)$	$0.606 (\pm 0.106)$	$0.718 (\pm 0.017)$	$0.735 (\pm 0.01)$
$1 \to 6$	$0.775 (\pm 0.091)$	$0.946 (\pm 0.0)$	$0.946 (\pm 0.0)$	$0.946 (\pm 0.0)$	$0.683 (\pm 0.076)$	$0.896 (\pm 0.069)$	$0.896 (\pm 0.069)$	$0.939 (\pm 0.006)$	0.947(±0.009)
$2 \to 7$	$0.527 (\pm 0.076)$	$0.473 (\pm 0.012)$	$0.473 (\pm 0.012)$	$0.475 (\pm 0.01)$	$0.534 (\pm 0.052)$	$0.472 (\pm 0.019)$	$0.555 (\pm 0.069)$	$0.472 (\pm 0.005)$	$0.558 (\pm 0.041)$
$3 \to 8$	$0.783 (\pm 0.015)$	0.971(±0.018	$0.971 (\pm 0.018)$	$0.969 (\pm 0.018)$	$0.97 (\pm 0.025)$	$0.789 (\pm 0.01)$	$0.789 (\pm 0.01)$	$0.96 (\pm 0.03)$	$0.987 (\pm 0.002)$
$4 \to 5$	$0.827 (\pm 0.039)$	$0.971 (\pm 0.013)$	$0.971 (\pm 0.013)$	$0.978 (\pm 0.002)$	$0.841 (\pm 0.1)$	$0.849 (\pm 0.074)$	$0.849 (\pm 0.074)$	$0.973 (\pm 0.004)$	$0.979 (\pm 0.006)$
平均值	$0.71 (\pm 0.063)$	$0.772 (\pm 0.009)$	$0.772 (\pm 0.009)$	$0.778 (\pm 0.007)$	$0.722 (\pm 0.075)$	$0.739 (\pm 0.048)$	$0.739 (\pm 0.066)$	$0.812 (\pm 0.012)$	$0.841 (\pm 0.014)$

Task	Deep-Coral
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.692 (\pm 0.026)$	$0.731 (\pm 0.028)$	$0.731 (\pm 0.028)$	$0.736 (\pm 0.024)$	$0.632 (\pm 0.024)$	$0.703 (\pm 0.038)$	$0.7 (\pm 0.038)$	$0.728 (\pm 0.019)$	$0.703 (\pm 0.024)$
$1 \to 6$	$0.862 (\pm 0.018)$	$0.893 (\pm 0.009)$	$0.893 (\pm 0.009)$	0.896(±0.004	$0.822 (\pm 0.104)$	$0.864 (\pm 0.017)$	$0.864 (\pm 0.017)$	$0.886 (\pm 0.009)$	$0.911 (\pm 0.017)$
$2 \to 7$	$0.509 (\pm 0.097)$	$0.454 (\pm 0.02)$	$0.454 (\pm 0.02)$	$0.457 (\pm 0.025)$	$0.51 (\pm 0.057)$	$0.499 (\pm 0.079)$	$0.539 (\pm 0.076)$	$0.46 (\pm 0.012)$	$0.565 (\pm 0.117)$
$3 \to 8$	$0.798 (\pm 0.016)$	$0.801 (\pm 0.004)$	0.801(±0.004	$0.803 (\pm 0.006)$	$0.793 (\pm 0.024)$	$0.799 (\pm 0.013)$	$0.802 (\pm 0.016)$	$0.812 (\pm 0.0)$	$0.822 (\pm 0.01)$
$4 \to 5$	$0.862 (\pm 0.036)$	$0.932 (\pm 0.025)$	$0.932 (\pm 0.025)$	$0.938 (\pm 0.022)$	$0.648 (\pm 0.108)$	$0.906 (\pm 0.027)$	$0.887 (\pm 0.035)$	$0.936 (\pm 0.016)$	$0.96 (\pm 0.01)$
Avg.	$0.745 (\pm 0.039)$	$0.762 (\pm 0.017)$	$0.762 (\pm 0.017)$	$0.766 (\pm 0.016)$	$0.681 (\pm 0.063)$	$0.754 (\pm 0.035)$	$0.758 (\pm 0.036)$	0.764(±0.011)	$0.792 (\pm 0.036)$

任务	深度珊瑚
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$0 \to 6$	$0.692 (\pm 0.026)$	$0.731 (\pm 0.028)$	$0.731 (\pm 0.028)$	$0.736 (\pm 0.024)$	$0.632 (\pm 0.024)$	$0.703 (\pm 0.038)$	$0.7 (\pm 0.038)$	$0.728 (\pm 0.019)$	$0.703 (\pm 0.024)$
$1 \to 6$	$0.862 (\pm 0.018)$	$0.893 (\pm 0.009)$	$0.893 (\pm 0.009)$	0.896(±0.004)	$0.822 (\pm 0.104)$	$0.864 (\pm 0.017)$	$0.864 (\pm 0.017)$	$0.886 (\pm 0.009)$	$0.911 (\pm 0.017)$
$2 \to 7$	$0.509 (\pm 0.097)$	$0.454 (\pm 0.02)$	$0.454 (\pm 0.02)$	$0.457 (\pm 0.025)$	$0.51 (\pm 0.057)$	$0.499 (\pm 0.079)$	$0.539 (\pm 0.076)$	$0.46 (\pm 0.012)$	$0.565 (\pm 0.117)$
$3 \to 8$	$0.798 (\pm 0.016)$	$0.801 (\pm 0.004)$	0.801(±0.004)	$0.803 (\pm 0.006)$	$0.793 (\pm 0.024)$	$0.799 (\pm 0.013)$	$0.802 (\pm 0.016)$	$0.812 (\pm 0.0)$	$0.822 (\pm 0.01)$
$4 \to 5$	$0.862 (\pm 0.036)$	$0.932 (\pm 0.025)$	$0.932 (\pm 0.025)$	$0.938 (\pm 0.022)$	$0.648 (\pm 0.108)$	$0.906 (\pm 0.027)$	$0.887 (\pm 0.035)$	$0.936 (\pm 0.016)$	$0.96 (\pm 0.01)$
平均值	$0.745 (\pm 0.039)$	$0.762 (\pm 0.017)$	$0.762 (\pm 0.017)$	$0.766 (\pm 0.016)$	$0.681 (\pm 0.063)$	$0.754 (\pm 0.035)$	$0.758 (\pm 0.036)$	0.764(±0.011)	$0.792 (\pm 0.036)$

Task	CDAN
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.717 (\pm 0.004)$	$0.483 (\pm 0.008)$	$0.483 (\pm 0.008)$	$0.481 (\pm 0.006)$	$0.699 (\pm 0.021)$	$0.692 (\pm 0.083)$	$0.692 (\pm 0.083)$	$0.76 (\pm 0.016)$	$0.718 (\pm 0.016)$
$1 \to 6$	$0.742 (\pm 0.09)$	$0.932 (\pm 0.006)$	$0.932 (\pm 0.006)$	$0.932 (\pm 0.01)$	$0.906 (\pm 0.021)$	$0.942 (\pm 0.011)$	$0.944 (\pm 0.013)$	$0.933 (\pm 0.0)$	$0.946 (\pm 0.0)$
$2 \to 7$	$0.554 (\pm 0.031)$	$0.522 (\pm 0.05)$	$0.522 (\pm 0.05)$	$0.56 (\pm 0.059)$	$0.58 (\pm 0.077)$	$0.561 (\pm 0.068)$	$0.561 (\pm 0.068)$	$0.531 (\pm 0.066)$	$0.624 (\pm 0.014)$
$3 \to 8$	$0.77 (\pm 0.069)$	$0.872 (\pm 0.09)$	$0.872 (\pm 0.09)$	$0.871 (\pm 0.091)$	$0.812 (\pm 0.007)$	$0.801 (\pm 0.004)$	$0.801 (\pm 0.004)$	$0.874 (\pm 0.076)$	$0.987 (\pm 0.006)$
$4 \to 5$	$0.859 (\pm 0.014)$	$0.978 (\pm 0.005)$	$0.978 (\pm 0.005)$	$0.979 (\pm 0.002)$	$0.828 (\pm 0.106)$	$0.875 (\pm 0.087)$	$0.875 (\pm 0.087)$	0.98(±0.0)	$0.982 (\pm 0.002)$
Avg.	$0.728 (\pm 0.042)$	$0.758 (\pm 0.032)$	$0.758 (\pm 0.032)$	$0.764 (\pm 0.034)$	$0.765 (\pm 0.046)$	$0.774 (\pm 0.051)$	$0.775 (\pm 0.051)$	$0.816 (\pm 0.031)$	$0.851 (\pm 0.008)$

任务	CDAN
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.717 (\pm 0.004)$	$0.483 (\pm 0.008)$	$0.483 (\pm 0.008)$	$0.481 (\pm 0.006)$	$0.699 (\pm 0.021)$	$0.692 (\pm 0.083)$	$0.692 (\pm 0.083)$	$0.76 (\pm 0.016)$	$0.718 (\pm 0.016)$
$1 \to 6$	$0.742 (\pm 0.09)$	$0.932 (\pm 0.006)$	$0.932 (\pm 0.006)$	$0.932 (\pm 0.01)$	$0.906 (\pm 0.021)$	$0.942 (\pm 0.011)$	$0.944 (\pm 0.013)$	$0.933 (\pm 0.0)$	$0.946 (\pm 0.0)$
$2 \to 7$	$0.554 (\pm 0.031)$	$0.522 (\pm 0.05)$	$0.522 (\pm 0.05)$	$0.56 (\pm 0.059)$	$0.58 (\pm 0.077)$	$0.561 (\pm 0.068)$	$0.561 (\pm 0.068)$	$0.531 (\pm 0.066)$	$0.624 (\pm 0.014)$
$3 \to 8$	$0.77 (\pm 0.069)$	$0.872 (\pm 0.09)$	$0.872 (\pm 0.09)$	$0.871 (\pm 0.091)$	$0.812 (\pm 0.007)$	$0.801 (\pm 0.004)$	$0.801 (\pm 0.004)$	$0.874 (\pm 0.076)$	$0.987 (\pm 0.006)$
$4 \to 5$	$0.859 (\pm 0.014)$	$0.978 (\pm 0.005)$	$0.978 (\pm 0.005)$	$0.979 (\pm 0.002)$	$0.828 (\pm 0.106)$	$0.875 (\pm 0.087)$	$0.875 (\pm 0.087)$	0.98(±0.0)	$0.982 (\pm 0.002)$
平均值	$0.728 (\pm 0.042)$	$0.758 (\pm 0.032)$	$0.758 (\pm 0.032)$	$0.764 (\pm 0.034)$	$0.765 (\pm 0.046)$	$0.774 (\pm 0.051)$	$0.775 (\pm 0.051)$	$0.816 (\pm 0.031)$	$0.851 (\pm 0.008)$

Task	DANN
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.704 (\pm 0.037)$	$0.489 (\pm 0.006)$	$0.489 (\pm 0.006)$	$0.49 (\pm 0.006)$	$0.585 (\pm 0.092)$	$0.724 (\pm 0.016)$	$0.724 (\pm 0.016)$	$0.722 (\pm 0.071)$	$0.711 (\pm 0.016)$
$1 \to 6$	$0.833 (\pm 0.036)$	$0.94 (\pm 0.002)$	$0.94 (\pm 0.002)$	$0.938 (\pm 0.004)$	$0.868 (\pm 0.142)$	$0.929 (\pm 0.004)$	$0.931 (\pm 0.006)$	$0.936 (\pm 0.002)$	$0.939 (\pm 0.002)$
$2 \to 7$	$0.591 (\pm 0.023)$	$0.496 (\pm 0.004)$	$0.496 (\pm 0.004)$	$0.493 (\pm 0.011)$	$0.557 (\pm 0.061$	$0.618 (\pm 0.029)$	$0.592 (\pm 0.022)$	$0.49 (\pm 0.003)$	$0.635 (\pm 0.025)$
$3 \to 8$	$0.809 (\pm 0.02)$	$0.966 (\pm 0.015$	$0.966 (\pm 0.015)$	$0.966 (\pm 0.015)$	$0.823 (\pm 0.069)$	$0.796 (\pm 0.03)$	$0.796 (\pm 0.03)$	$0.964 (\pm 0.006)$	$0.983 (\pm 0.002)$
$4 \to 5$	$0.846 (\pm 0.022)$	$0.98 (\pm 0.0)$	$0.98 (\pm 0.0)$	$0.98 (\pm 0.0)$	$0.779 (\pm 0.138)$	$0.922 (\pm 0.078)$	$0.922 (\pm 0.078)$	$0.98 (\pm 0.0)$	$0.98 (\pm 0.0)$
Avg.	$0.757 (\pm 0.027)$	$0.774 (\pm 0.006)$	$0.774 (\pm 0.006)$	$0.773 (\pm 0.007)$	$0.722 (\pm 0.1)$	$0.798 (\pm 0.031)$	$0.793 (\pm 0.031)$	$0.818 (\pm 0.016)$	$0.85 (\pm 0.009)$

任务	DANN
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.704 (\pm 0.037)$	$0.489 (\pm 0.006)$	$0.489 (\pm 0.006)$	$0.49 (\pm 0.006)$	$0.585 (\pm 0.092)$	$0.724 (\pm 0.016)$	$0.724 (\pm 0.016)$	$0.722 (\pm 0.071)$	$0.711 (\pm 0.016)$
$1 \to 6$	$0.833 (\pm 0.036)$	$0.94 (\pm 0.002)$	$0.94 (\pm 0.002)$	$0.938 (\pm 0.004)$	$0.868 (\pm 0.142)$	$0.929 (\pm 0.004)$	$0.931 (\pm 0.006)$	$0.936 (\pm 0.002)$	$0.939 (\pm 0.002)$
$2 \to 7$	$0.591 (\pm 0.023)$	$0.496 (\pm 0.004)$	$0.496 (\pm 0.004)$	$0.493 (\pm 0.011)$	$0.557 (\pm 0.061$	$0.618 (\pm 0.029)$	$0.592 (\pm 0.022)$	$0.49 (\pm 0.003)$	$0.635 (\pm 0.025)$
$3 \to 8$	$0.809 (\pm 0.02)$	$0.966 (\pm 0.015$	$0.966 (\pm 0.015)$	$0.966 (\pm 0.015)$	$0.823 (\pm 0.069)$	$0.796 (\pm 0.03)$	$0.796 (\pm 0.03)$	$0.964 (\pm 0.006)$	$0.983 (\pm 0.002)$
$4 \to 5$	$0.846 (\pm 0.022)$	$0.98 (\pm 0.0)$	$0.98 (\pm 0.0)$	$0.98 (\pm 0.0)$	$0.779 (\pm 0.138)$	$0.922 (\pm 0.078)$	$0.922 (\pm 0.078)$	$0.98 (\pm 0.0)$	$0.98 (\pm 0.0)$
平均	$0.757 (\pm 0.027)$	$0.774 (\pm 0.006)$	$0.774 (\pm 0.006)$	$0.773 (\pm 0.007)$	$0.722 (\pm 0.1)$	$0.798 (\pm 0.031)$	$0.793 (\pm 0.031)$	$0.818 (\pm 0.016)$	$0.85 (\pm 0.009)$

Task	DSAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$0 \to 6$	$0.653 (\pm 0.033)$	$0.607 (\pm 0.057)$	$0.607 (\pm 0.057)$	$0.635 (\pm 0.119)$	$0.597 (\pm 0.075)$	$0.572 (\pm 0.103)$	$0.471 (\pm 0.259)$	$0.747 (\pm 0.027)$	$0.664 (\pm 0.027)$
$1 \to 6$	$0.806 (\pm 0.083)$	$0.922 (\pm 0.002)$	$0.922 (\pm 0.002)$	$0.925 (\pm 0.004$	$0.821 (\pm 0.105)$	$0.908 (\pm 0.065)$	$0.908 (\pm 0.065)$	$0.929 (\pm 0.0)$	$0.94 (\pm 0.005)$
$2 \to 7$	$0.49 (\pm 0.053)$	$0.496 (\pm 0.004$	$0.496 (\pm 0.004)$	$0.494 (\pm 0.003)$	$0.485 (\pm 0.005)$	$0.482 (\pm 0.012)$	$0.49 (\pm 0.003)$	$0.496 (\pm 0.0)$	$0.586 (\pm 0.083)$
$3 \to 8$	$0.797 (\pm 0.016)$	$0.979 (\pm 0.002$	0.979(±0.002)	$0.977 (\pm 0.004)$	$0.855 (\pm 0.097)$	$0.816 (\pm 0.01)$	$0.639 (\pm 0.454)$	$0.971 (\pm 0.005)$	$0.982 (\pm 0.01)$
$4 \to 5$	$0.861 (\pm 0.043)$	$0.98 (\pm 0.008)$	$0.98 (\pm 0.008)$	0.979/±0.006	$0.863 (\pm 0.085)$	$0.928 (\pm 0.08)$	$0.471 (\pm 0.438)$	$0.98 (\pm 0.0)$	$0.982 (\pm 0.005)$
Avg.	$0.721 (\pm 0.046)$	$0.797 (\pm 0.015)$	$0.797 (\pm 0.015)$	$0.802 (\pm 0.027)$	$0.724 (\pm 0.073)$	$0.741 (\pm 0.054)$	$0.596 (\pm 0.244)$	$0.825 (\pm 0.006)$	$0.831 (\pm 0.026)$

任务	DSAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$0 \to 6$	$0.653 (\pm 0.033)$	$0.607 (\pm 0.057)$	$0.607 (\pm 0.057)$	$0.635 (\pm 0.119)$	$0.597 (\pm 0.075)$	$0.572 (\pm 0.103)$	$0.471 (\pm 0.259)$	$0.747 (\pm 0.027)$	$0.664 (\pm 0.027)$
$1 \to 6$	$0.806 (\pm 0.083)$	$0.922 (\pm 0.002)$	$0.922 (\pm 0.002)$	$0.925 (\pm 0.004$	$0.821 (\pm 0.105)$	$0.908 (\pm 0.065)$	$0.908 (\pm 0.065)$	$0.929 (\pm 0.0)$	$0.94 (\pm 0.005)$
$2 \to 7$	$0.49 (\pm 0.053)$	$0.496 (\pm 0.004$	$0.496 (\pm 0.004)$	$0.494 (\pm 0.003)$	$0.485 (\pm 0.005)$	$0.482 (\pm 0.012)$	$0.49 (\pm 0.003)$	$0.496 (\pm 0.0)$	$0.586 (\pm 0.083)$
$3 \to 8$	$0.797 (\pm 0.016)$	$0.979 (\pm 0.002$	0.979（±0.002）	$0.977 (\pm 0.004)$	$0.855 (\pm 0.097)$	$0.816 (\pm 0.01)$	$0.639 (\pm 0.454)$	$0.971 (\pm 0.005)$	$0.982 (\pm 0.01)$
$4 \to 5$	$0.861 (\pm 0.043)$	$0.98 (\pm 0.008)$	$0.98 (\pm 0.008)$	0.979/±0.006	$0.863 (\pm 0.085)$	$0.928 (\pm 0.08)$	$0.471 (\pm 0.438)$	$0.98 (\pm 0.0)$	$0.982 (\pm 0.005)$
平均值	$0.721 (\pm 0.046)$	$0.797 (\pm 0.015)$	$0.797 (\pm 0.015)$	$0.802 (\pm 0.027)$	$0.724 (\pm 0.073)$	$0.741 (\pm 0.054)$	$0.596 (\pm 0.244)$	$0.825 (\pm 0.006)$	$0.831 (\pm 0.026)$

Task	HoMM
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.733 (\pm 0.058)$	$0.744 (\pm 0.038)$	$0.744 (\pm 0.038)$	$0.756 (\pm 0.051)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.058)$	$0.733 (\pm 0.058)$	$0.711 (\pm 0.019)$	$0.767 (\pm 0.033)$
$20 \to 30$	$0.853 (\pm 0.022)$	$0.814 (\pm 0.022)$	$0.814 (\pm 0.022)$	$0.801 (\pm 0.011)$	$0.827 (\pm 0.0)$	$0.853 (\pm 0.022)$	$0.853 (\pm 0.022)$	$0.814 (\pm 0.011)$	$0.853 (\pm 0.022)$
$35 \to 31$	$0.579 (\pm 0.055)$	0.77(±0.014	$0.77 (\pm 0.014)$	$0.77 (\pm 0.014)$	$0.722 (\pm 0.05)$	$0.579 (\pm 0.055)$	$0.579 (\pm 0.055)$	$0.746 (\pm 0.027)$	$0.77 (\pm 0.014)$
$6 \to 19$	$0.889 (\pm 0.009)$	$0.864 (\pm 0.052)$	$0.864 (\pm 0.052)$	$0.823 (\pm 0.035)$	0.884(±0.017)	$0.889 (\pm 0.009)$	$0.889 (\pm 0.009)$	$0.823 (\pm 0.035)$	$0.894 (\pm 0.026)$
$7 \to 18$	$0.711 (\pm 0.066)$	$0.497 (\pm 0.011)$	$0.497 (\pm 0.011)$	$0.547 (\pm 0.068)$	$0.711 (\pm 0.029)$	0.711(±0.066	$0.648 (\pm 0.123)$	$0.547 (\pm 0.0)$	$0.711 (\pm 0.066)$
Avg.	$0.753 (\pm 0.042)$	$0.738 (\pm 0.028)$	$0.738 (\pm 0.028)$	$0.739 (\pm 0.036)$	$0.775 (\pm 0.026)$	$0.753 (\pm 0.042)$	$0.74 (\pm 0.053)$	$0.728 (\pm 0.019)$	$0.799 (\pm 0.032)$

任务	HoMM
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.733 (\pm 0.058)$	$0.744 (\pm 0.038)$	$0.744 (\pm 0.038)$	$0.756 (\pm 0.051)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.058)$	$0.733 (\pm 0.058)$	$0.711 (\pm 0.019)$	$0.767 (\pm 0.033)$
$20 \to 30$	$0.853 (\pm 0.022)$	$0.814 (\pm 0.022)$	$0.814 (\pm 0.022)$	$0.801 (\pm 0.011)$	$0.827 (\pm 0.0)$	$0.853 (\pm 0.022)$	$0.853 (\pm 0.022)$	$0.814 (\pm 0.011)$	$0.853 (\pm 0.022)$
$35 \to 31$	$0.579 (\pm 0.055)$	0.77(±0.014	$0.77 (\pm 0.014)$	$0.77 (\pm 0.014)$	$0.722 (\pm 0.05)$	$0.579 (\pm 0.055)$	$0.579 (\pm 0.055)$	$0.746 (\pm 0.027)$	$0.77 (\pm 0.014)$
$6 \to 19$	$0.889 (\pm 0.009)$	$0.864 (\pm 0.052)$	$0.864 (\pm 0.052)$	$0.823 (\pm 0.035)$	0.884(±0.017)	$0.889 (\pm 0.009)$	$0.889 (\pm 0.009)$	$0.823 (\pm 0.035)$	$0.894 (\pm 0.026)$
$7 \to 18$	$0.711 (\pm 0.066)$	$0.497 (\pm 0.011)$	$0.497 (\pm 0.011)$	$0.547 (\pm 0.068)$	$0.711 (\pm 0.029)$	0.711(±0.066	$0.648 (\pm 0.123)$	$0.547 (\pm 0.0)$	$0.711 (\pm 0.066)$
平均值	$0.753 (\pm 0.042)$	$0.738 (\pm 0.028)$	$0.738 (\pm 0.028)$	$0.739 (\pm 0.036)$	$0.775 (\pm 0.026)$	$0.753 (\pm 0.042)$	$0.74 (\pm 0.053)$	$0.728 (\pm 0.019)$	$0.799 (\pm 0.032)$

Task	AdvSKM
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.711 (\pm 0.019)$	$0.744 (\pm 0.038)$	$0.744 (\pm 0.038)$	$0.744 (\pm 0.038)$	$0.767 (\pm 0.033)$	$0.711 (\pm 0.019)$	$0.711 (\pm 0.019)$	$0.744 (\pm 0.019)$	$0.778 (\pm 0.096)$
$20 \to 30$	$0.872 (\pm 0.029)$	$0.885 (\pm 0.033)$	$0.885 (\pm 0.033)$	$0.885 (\pm 0.033)$	$0.872 (\pm 0.059)$	$0.872 (\pm 0.029)$	$0.872 (\pm 0.029$	$0.853 (\pm 0.029)$	$0.885 (\pm 0.033)$
$35 \to 31$	$0.619 (\pm 0.041)$	$0.698 (\pm 0.027)$	$0.698 (\pm 0.027)$	$0.69 (\pm 0.082)$	$0.675 (\pm 0.096)$	$0.619 (\pm 0.041)$	$0.619 (\pm 0.041)$	$0.698 (\pm 0.027)$	$0.714 (\pm 0.024)$
$6 \to 19$	$0.818 (\pm 0.131)$	$0.894 (\pm 0.0)$	$0.894 (\pm 0.0)$	$0.869 (\pm 0.044)$	$0.833 (\pm 0.052)$	0.818(±0.131)	$0.818 (\pm 0.131)$	$0.874 (\pm 0.009)$	$0.894 (\pm 0.0)$
$7 \to 18$	$0.717 (\pm 0.05)$	$0.686 (\pm 0.058)$	$0.686 (\pm 0.058)$	$0.704 (\pm 0.039)$	$0.566 (\pm 0.068)$	$0.717 (\pm 0.05)$	$0.717 (\pm 0.05)$	$0.717 (\pm 0.019)$	$0.736 (\pm 0.086)$
Avg.	$0.747 (\pm 0.054)$	$0.781 (\pm 0.031)$	$0.781 (\pm 0.031)$	$0.779 (\pm 0.047)$	$0.742 (\pm 0.062)$	$0.747 (\pm 0.054)$	$0.747 (\pm 0.054)$	$0.777 (\pm 0.021)$	$0.801 (\pm 0.048)$

任务	AdvSKM
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.711 (\pm 0.019)$	$0.744 (\pm 0.038)$	$0.744 (\pm 0.038)$	$0.744 (\pm 0.038)$	$0.767 (\pm 0.033)$	$0.711 (\pm 0.019)$	$0.711 (\pm 0.019)$	$0.744 (\pm 0.019)$	$0.778 (\pm 0.096)$
$20 \to 30$	$0.872 (\pm 0.029)$	$0.885 (\pm 0.033)$	$0.885 (\pm 0.033)$	$0.885 (\pm 0.033)$	$0.872 (\pm 0.059)$	$0.872 (\pm 0.029)$	$0.872 (\pm 0.029$	$0.853 (\pm 0.029)$	$0.885 (\pm 0.033)$
$35 \to 31$	$0.619 (\pm 0.041)$	$0.698 (\pm 0.027)$	$0.698 (\pm 0.027)$	$0.69 (\pm 0.082)$	$0.675 (\pm 0.096)$	$0.619 (\pm 0.041)$	$0.619 (\pm 0.041)$	$0.698 (\pm 0.027)$	$0.714 (\pm 0.024)$
$6 \to 19$	$0.818 (\pm 0.131)$	$0.894 (\pm 0.0)$	$0.894 (\pm 0.0)$	$0.869 (\pm 0.044)$	$0.833 (\pm 0.052)$	0.818(±0.131)	$0.818 (\pm 0.131)$	$0.874 (\pm 0.009)$	$0.894 (\pm 0.0)$
$7 \to 18$	$0.717 (\pm 0.05)$	$0.686 (\pm 0.058)$	$0.686 (\pm 0.058)$	$0.704 (\pm 0.039)$	$0.566 (\pm 0.068)$	$0.717 (\pm 0.05)$	$0.717 (\pm 0.05)$	$0.717 (\pm 0.019)$	$0.736 (\pm 0.086)$
平均值	$0.747 (\pm 0.054)$	$0.781 (\pm 0.031)$	$0.781 (\pm 0.031)$	$0.779 (\pm 0.047)$	$0.742 (\pm 0.062)$	$0.747 (\pm 0.054)$	$0.747 (\pm 0.054)$	$0.777 (\pm 0.021)$	$0.801 (\pm 0.048)$

Task	DIRT
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	1WV	DEV	IWA (ours)	TB
$18 \to 23$	$0.711 (\pm 0.038)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.0)$	$0.711 (\pm 0.019)$	$0.711 (\pm 0.038)$	$0.711 (\pm 0.038)$	$0.733 (\pm 0.0)$	$0.744 (\pm 0.019)$
$20 \to 30$	$0.872 (\pm 0.011)$	$0.846 (\pm 0.0)$	$0.846 (\pm 0.0)$	$0.846 (\pm 0.0)$	0.891(±0.04	$0.872 (\pm 0.011)$	$0.859 (\pm 0.029)$	$0.84 (\pm 0.011)$	$0.917 (\pm 0.011)$
$35 \to 31$	$0.492 (\pm 0.036)$	$0.746 (\pm 0.036)$	$0.746 (\pm 0.036)$	$0.746 (\pm 0.036)$	$0.603 (\pm 0.107)$	$0.492 (\pm 0.036)$	0.77(±0.069	$0.746 (\pm 0.014)$	$0.778 (\pm 0.027)$
$6 \to 19$	$0.879 (\pm 0.026)$	$0.833 (\pm 0.052)$	$0.833 (\pm 0.052)$	$0.833 / \pm {0.052}^{\circ}$	0.869(±0.023	$0.879 (\pm 0.026)$	$0.879 (\pm 0.026)$	$0.838 (\pm 0.049)$	$0.889 (\pm 0.009)$
$7 \to 18$	$0.736 (\pm 0.082)$	0.824(±0.011)	$0.824 (\pm 0.011)$	$0.824 (\pm 0.011)$	$0.704 (\pm 0.093)$	$0.736 (\pm 0.082)$	$0.767 (\pm 0.109)$	$0.83 (\pm 0.0)$	$0.83 (\pm 0.0)$
Avg.	$0.738 (\pm 0.039)$	$0.797 (\pm 0.027)$	$0.797 (\pm 0.027)$	$0.797 (\pm 0.02)$	$0.756 (\pm 0.057)$	$0.738 (\pm 0.039)$	$0.797 (\pm 0.054)$	$0.798 (\pm 0.015)$	$0.832 (\pm 0.013)$

任务	DIRT
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	1WV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.711 (\pm 0.038)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.0)$	$0.711 (\pm 0.019)$	$0.711 (\pm 0.038)$	$0.711 (\pm 0.038)$	$0.733 (\pm 0.0)$	$0.744 (\pm 0.019)$
$20 \to 30$	$0.872 (\pm 0.011)$	$0.846 (\pm 0.0)$	$0.846 (\pm 0.0)$	$0.846 (\pm 0.0)$	0.891(±0.04	$0.872 (\pm 0.011)$	$0.859 (\pm 0.029)$	$0.84 (\pm 0.011)$	$0.917 (\pm 0.011)$
$35 \to 31$	$0.492 (\pm 0.036)$	$0.746 (\pm 0.036)$	$0.746 (\pm 0.036)$	$0.746 (\pm 0.036)$	$0.603 (\pm 0.107)$	$0.492 (\pm 0.036)$	0.77(±0.069	$0.746 (\pm 0.014)$	$0.778 (\pm 0.027)$
$6 \to 19$	$0.879 (\pm 0.026)$	$0.833 (\pm 0.052)$	$0.833 (\pm 0.052)$	$0.833 / \pm {0.052}^{\circ}$	0.869(±0.023	$0.879 (\pm 0.026)$	$0.879 (\pm 0.026)$	$0.838 (\pm 0.049)$	$0.889 (\pm 0.009)$
$7 \to 18$	$0.736 (\pm 0.082)$	0.824(±0.011)	$0.824 (\pm 0.011)$	$0.824 (\pm 0.011)$	$0.704 (\pm 0.093)$	$0.736 (\pm 0.082)$	$0.767 (\pm 0.109)$	$0.83 (\pm 0.0)$	$0.83 (\pm 0.0)$
平均值	$0.738 (\pm 0.039)$	$0.797 (\pm 0.027)$	$0.797 (\pm 0.027)$	$0.797 (\pm 0.02)$	$0.756 (\pm 0.057)$	$0.738 (\pm 0.039)$	$0.797 (\pm 0.054)$	$0.798 (\pm 0.015)$	$0.832 (\pm 0.013)$

Task	DDC
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.7 (\pm 0.067)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.033)$	$0.767 (\pm 0.033)$	0.7(±0.0)	$0.7 (\pm 0.067)$	0.7(±0.067)	$0.744 (\pm 0.019)$	$0.767 (\pm 0.0)$
$20 \to 30$	$0.833 (\pm 0.022)$	$0.865 (\pm 0.0)$	$0.865 (\pm 0.0)$	$0.859 (\pm 0.011)$	$0.833 (\pm 0.029)$	$0.833 (\pm 0.022)$	$0.833 (\pm 0.022)$	$0.84 (\pm 0.011)$	$0.885 (\pm 0.019)$
$35 \to 31$	$0.54 (\pm 0.036)$	$0.722 (\pm 0.06)$	$0.722 (\pm 0.06)$	0.746(±0.069	$0.635 (\pm 0.096)$	$0.54 (\pm 0.036)$	$0.54 (\pm 0.036)$	$0.746 (\pm 0.027$	$0.73 (\pm 0.027)$
$6 \to 19$	$0.884 (\pm 0.017)$	0.884(±0.017)	$0.884 (\pm 0.017)$	$0.859 (\pm 0.032)$	$0.869 (\pm 0.044)$	$0.884 (\pm 0.017)$	$0.884 (\pm 0.017)$	$0.869 (\pm 0.023)$	$0.884 (\pm 0.017)$
$7 \to 18$	$0.748 (\pm 0.093)$	$0.692 (\pm 0.039)$	$0.692 (\pm 0.039$	0.704(±0.029)	$0.648 (\pm 0.079)$	$0.748 (\pm 0.093)$	$0.748 (\pm 0.093)$	$0.711 (\pm 0.011)$	$0.748 (\pm 0.093)$
Avg.	$0.741 (\pm 0.047)$	$0.779 (\pm 0.03)$	$0.779 (\pm 0.03)$	$0.787 (\pm 0.035)$	$0.737 (\pm 0.05)$	$0.741 (\pm 0.047)$	$0.741 (\pm 0.047)$	$0.782 (\pm 0.018)$	$0.803 (\pm 0.031)$

任务	DDC
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.7 (\pm 0.067)$	$0.733 (\pm 0.033)$	$0.733 (\pm 0.033)$	$0.767 (\pm 0.033)$	0.7(±0.0)	$0.7 (\pm 0.067)$	0.7(±0.067)	$0.744 (\pm 0.019)$	$0.767 (\pm 0.0)$
$20 \to 30$	$0.833 (\pm 0.022)$	$0.865 (\pm 0.0)$	$0.865 (\pm 0.0)$	$0.859 (\pm 0.011)$	$0.833 (\pm 0.029)$	$0.833 (\pm 0.022)$	$0.833 (\pm 0.022)$	$0.84 (\pm 0.011)$	$0.885 (\pm 0.019)$
$35 \to 31$	$0.54 (\pm 0.036)$	$0.722 (\pm 0.06)$	$0.722 (\pm 0.06)$	0.746(±0.069)	$0.635 (\pm 0.096)$	$0.54 (\pm 0.036)$	$0.54 (\pm 0.036)$	$0.746 (\pm 0.027$	$0.73 (\pm 0.027)$
$6 \to 19$	$0.884 (\pm 0.017)$	0.884(±0.017)	$0.884 (\pm 0.017)$	$0.859 (\pm 0.032)$	$0.869 (\pm 0.044)$	$0.884 (\pm 0.017)$	$0.884 (\pm 0.017)$	$0.869 (\pm 0.023)$	$0.884 (\pm 0.017)$
$7 \to 18$	$0.748 (\pm 0.093)$	$0.692 (\pm 0.039)$	$0.692 (\pm 0.039$	0.704(±0.029)	$0.648 (\pm 0.079)$	$0.748 (\pm 0.093)$	$0.748 (\pm 0.093)$	$0.711 (\pm 0.011)$	$0.748 (\pm 0.093)$
平均值	$0.741 (\pm 0.047)$	$0.779 (\pm 0.03)$	$0.779 (\pm 0.03)$	$0.787 (\pm 0.035)$	$0.737 (\pm 0.05)$	$0.741 (\pm 0.047)$	$0.741 (\pm 0.047)$	$0.782 (\pm 0.018)$	$0.803 (\pm 0.031)$

Task	CMD
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.711 (\pm 0.038)$	$0.678 (\pm 0.019)$	$0.678 (\pm 0.019)$	$0.644 (\pm 0.069)$	$0.689 (\pm 0.107)$	$0.689 (\pm 0.107)$	$0.622 (\pm 0.069)$	0.7(±0.0)	$0.756 (\pm 0.038)$
$20 \to 30$	$0.853 (\pm 0.011)$	$0.859 (\pm 0.022)$	$0.859 (\pm 0.022)$	0.897(±0.029	$0.885 (\pm 0.051)$	$0.853 (\pm 0.011)$	$0.859 (\pm 0.011)$	$0.904 (\pm 0.0)$	$0.91 (\pm 0.022)$
$35 \to 31$	$0.683 (\pm 0.014)$	$0.722 (\pm 0.014)$	$0.722 (\pm 0.014)$	$0.722 (\pm 0.014)$	$0.659 (\pm 0.014)$	$0.683 (\pm 0.014)$	$0.69 (\pm 0.024)$	$0.722 (\pm 0.027)$	$0.77 (\pm 0.014)$
$6 \to 19$	$0.742 (\pm 0.084)$	$0.798 (\pm 0.009)$	$0.798 (\pm 0.009)$	$0.803 (\pm 0.015)$	$0.833 (\pm 0.12)$	$0.783 (\pm 0.118)$	$0.682 (\pm 0.124)$	$0.798 (\pm 0.035)$	$0.813 (\pm 0.12)$
$7 \to 18$	$0.56 (\pm 0.044)$	$0.767 (\pm 0.058)$	$0.767 (\pm 0.058)$	$0.767 (\pm 0.058)$	$0.572 (\pm 0.061)$	$0.56 (\pm 0.044)$	$0.579 (\pm 0.066)$	$0.742 (\pm 0.029)$	$0.736 (\pm 0.058)$
Avg.	$0.71 (\pm 0.038)$	$0.765 (\pm 0.024)$	$0.765 (\pm 0.024)$	$0.767 (\pm 0.037)$	$0.728 (\pm 0.071)$	$0.713 (\pm 0.059)$	$0.686 (\pm 0.059)$	$0.773 (\pm 0.018)$	$0.797 (\pm 0.05)$

任务	命令
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.711 (\pm 0.038)$	$0.678 (\pm 0.019)$	$0.678 (\pm 0.019)$	$0.644 (\pm 0.069)$	$0.689 (\pm 0.107)$	$0.689 (\pm 0.107)$	$0.622 (\pm 0.069)$	0.7（±0.0）	$0.756 (\pm 0.038)$
$20 \to 30$	$0.853 (\pm 0.011)$	$0.859 (\pm 0.022)$	$0.859 (\pm 0.022)$	0.897（±0.029）	$0.885 (\pm 0.051)$	$0.853 (\pm 0.011)$	$0.859 (\pm 0.011)$	$0.904 (\pm 0.0)$	$0.91 (\pm 0.022)$
$35 \to 31$	$0.683 (\pm 0.014)$	$0.722 (\pm 0.014)$	$0.722 (\pm 0.014)$	$0.722 (\pm 0.014)$	$0.659 (\pm 0.014)$	$0.683 (\pm 0.014)$	$0.69 (\pm 0.024)$	$0.722 (\pm 0.027)$	$0.77 (\pm 0.014)$
$6 \to 19$	$0.742 (\pm 0.084)$	$0.798 (\pm 0.009)$	$0.798 (\pm 0.009)$	$0.803 (\pm 0.015)$	$0.833 (\pm 0.12)$	$0.783 (\pm 0.118)$	$0.682 (\pm 0.124)$	$0.798 (\pm 0.035)$	$0.813 (\pm 0.12)$
$7 \to 18$	$0.56 (\pm 0.044)$	$0.767 (\pm 0.058)$	$0.767 (\pm 0.058)$	$0.767 (\pm 0.058)$	$0.572 (\pm 0.061)$	$0.56 (\pm 0.044)$	$0.579 (\pm 0.066)$	$0.742 (\pm 0.029)$	$0.736 (\pm 0.058)$
平均值	$0.71 (\pm 0.038)$	$0.765 (\pm 0.024)$	$0.765 (\pm 0.024)$	$0.767 (\pm 0.037)$	$0.728 (\pm 0.071)$	$0.713 (\pm 0.059)$	$0.686 (\pm 0.059)$	$0.773 (\pm 0.018)$	$0.797 (\pm 0.05)$

Task	MMDA
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.756 (\pm 0.051)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	$0.756 (\pm 0.019)$	$0.756 (\pm 0.051)$	$0.756 (\pm 0.051)$	0.8(±0.0)	$0.889 (\pm 0.019)$
$20 \to 30$	$0.872 (\pm 0.011)$	$0.878 (\pm 0.029)$	$0.878 (\pm 0.029)$	$0.872 (\pm 0.029)$	$0.814 (\pm 0.029)$	$0.872 (\pm 0.011)$	$0.872 (\pm 0.011)$	$0.878 (\pm 0.029)$	$0.891 (\pm 0.04)$
$35 \to 31$	$0.571 (\pm 0.024)$	$0.786 (\pm 0.0)$	$0.786 (\pm 0.0)$	$0.754 (\pm 0.036)$	$0.683 (\pm 0.107)$	$0.571 (\pm 0.024)$	$0.571 (\pm 0.024)$	$0.77 (\pm 0.014)$	$0.746 (\pm 0.0)$
$6 \to 19$	$0.879 (\pm 0.015)$	$0.843 (\pm 0.087)$	$0.843 (\pm 0.087)$	$0.747 (\pm 0.009)$	$0.864 (\pm 0.04)$	$0.879 (\pm 0.015)$	$0.879 (\pm 0.015)$	$0.854 (\pm 0.009)$	$0.909 (\pm 0.015)$
$7 \to 18$	$0.717 (\pm 0.05)$	$0.585 (\pm 0.019)$	$0.585 (\pm 0.019)$	$0.585 (\pm 0.019)$	$0.654 (\pm 0.143)$	$0.717 (\pm 0.05)$	$0.673 (\pm 0.022)$	$0.648 (\pm 0.011)$	$0.723 (\pm 0.029)$
Avg.	$0.759 (\pm 0.03)$	$0.772 (\pm 0.034)$	$0.772 (\pm 0.034)$	$0.745 (\pm 0.025)$	$0.754 (\pm 0.068)$	$0.759 (\pm 0.03)$	$0.75 (\pm 0.025)$	0.79(±0.013)	$0.832 (\pm 0.021)$

任务	MMDA
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$18 \to 23$	$0.756 (\pm 0.051)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	$0.756 (\pm 0.019)$	$0.756 (\pm 0.051)$	$0.756 (\pm 0.051)$	0.8(±0.0)	$0.889 (\pm 0.019)$
$20 \to 30$	$0.872 (\pm 0.011)$	$0.878 (\pm 0.029)$	$0.878 (\pm 0.029)$	$0.872 (\pm 0.029)$	$0.814 (\pm 0.029)$	$0.872 (\pm 0.011)$	$0.872 (\pm 0.011)$	$0.878 (\pm 0.029)$	$0.891 (\pm 0.04)$
$35 \to 31$	$0.571 (\pm 0.024)$	$0.786 (\pm 0.0)$	$0.786 (\pm 0.0)$	$0.754 (\pm 0.036)$	$0.683 (\pm 0.107)$	$0.571 (\pm 0.024)$	$0.571 (\pm 0.024)$	$0.77 (\pm 0.014)$	$0.746 (\pm 0.0)$
$6 \to 19$	$0.879 (\pm 0.015)$	$0.843 (\pm 0.087)$	$0.843 (\pm 0.087)$	$0.747 (\pm 0.009)$	$0.864 (\pm 0.04)$	$0.879 (\pm 0.015)$	$0.879 (\pm 0.015)$	$0.854 (\pm 0.009)$	$0.909 (\pm 0.015)$
$7 \to 18$	$0.717 (\pm 0.05)$	$0.585 (\pm 0.019)$	$0.585 (\pm 0.019)$	$0.585 (\pm 0.019)$	$0.654 (\pm 0.143)$	$0.717 (\pm 0.05)$	$0.673 (\pm 0.022)$	$0.648 (\pm 0.011)$	$0.723 (\pm 0.029)$
平均值	$0.759 (\pm 0.03)$	$0.772 (\pm 0.034)$	$0.772 (\pm 0.034)$	$0.745 (\pm 0.025)$	$0.754 (\pm 0.068)$	$0.759 (\pm 0.03)$	$0.75 (\pm 0.025)$	0.79(±0.013)	$0.832 (\pm 0.021)$

Task	CoDATS
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.689 (\pm 0.051)$	$0.722 (\pm 0.069)$	$0.722 (\pm 0.069)$	$0.738 (\pm 0.058)$	$0.667 (\pm 0.033)$	$0.678 (\pm 0.077)$	$0.7 (\pm 0.067)$	$0.756 (\pm 0.019)$	$0.756 (\pm 0.019)$
$20 \to 30$	$0.846 (\pm 0.069)$	$0.891 (\pm 0.029)$	$0.891 (\pm 0.029)$	$0.885 (\pm 0.038)$	$0.776 (\pm 0.022)$	$0.846 (\pm 0.069)$	$0.846 (\pm 0.069)$	$0.904 (\pm 0.019)$	$0.904 (\pm 0.051)$
$35 \to 31$	$0.619 (\pm 0.095)$	$0.738 (\pm 0.048)$	$0.738 (\pm 0.048)$	$0.722 (\pm 0.014)$	$0.643 (\pm 0.124)$	$0.619 (\pm 0.095)$	$0.73 (\pm 0.036)$	$0.714 (\pm 0.063)$	$0.786 (\pm 0.024)$
$6 \to 19$	$0.727 (\pm 0.069)$	$0.899 (\pm 0.038)$	$0.899 (\pm 0.038)$	$0.864 (\pm 0.069)$	$0.742 (\pm 0.076)$	$0.727 (\pm 0.069)$	$0.727 (\pm 0.069)$	$0.899 (\pm 0.032)$	$0.924 (\pm 0.08)$
$7 \to 18$	$0.673 (\pm 0.071)$	$0.535 (\pm 0.076)$	$0.535 (\pm 0.076)$	$0.553 (\pm 0.066)$	$0.585 (\pm 0.098$	$0.673 (\pm 0.071)$	$0.673 (\pm 0.071$	$0.547 (\pm 0.057)$	$0.692 (\pm 0.142)$
Avg.	$0.711 (\pm 0.071)$	$0.757 (\pm 0.052)$	$0.757 (\pm 0.052)$	$0.751 (\pm 0.049)$	$0.682 (\pm 0.071)$	$0.709 (\pm 0.076)$	$0.735 (\pm 0.063)$	$0.764 (\pm 0.038)$	$0.812 (\pm 0.063)$

任务	CoDATS
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.689 (\pm 0.051)$	$0.722 (\pm 0.069)$	$0.722 (\pm 0.069)$	$0.738 (\pm 0.058)$	$0.667 (\pm 0.033)$	$0.678 (\pm 0.077)$	$0.7 (\pm 0.067)$	$0.756 (\pm 0.019)$	$0.756 (\pm 0.019)$
$20 \to 30$	$0.846 (\pm 0.069)$	$0.891 (\pm 0.029)$	$0.891 (\pm 0.029)$	$0.885 (\pm 0.038)$	$0.776 (\pm 0.022)$	$0.846 (\pm 0.069)$	$0.846 (\pm 0.069)$	$0.904 (\pm 0.019)$	$0.904 (\pm 0.051)$
$35 \to 31$	$0.619 (\pm 0.095)$	$0.738 (\pm 0.048)$	$0.738 (\pm 0.048)$	$0.722 (\pm 0.014)$	$0.643 (\pm 0.124)$	$0.619 (\pm 0.095)$	$0.73 (\pm 0.036)$	$0.714 (\pm 0.063)$	$0.786 (\pm 0.024)$
$6 \to 19$	$0.727 (\pm 0.069)$	$0.899 (\pm 0.038)$	$0.899 (\pm 0.038)$	$0.864 (\pm 0.069)$	$0.742 (\pm 0.076)$	$0.727 (\pm 0.069)$	$0.727 (\pm 0.069)$	$0.899 (\pm 0.032)$	$0.924 (\pm 0.08)$
$7 \to 18$	$0.673 (\pm 0.071)$	$0.535 (\pm 0.076)$	$0.535 (\pm 0.076)$	$0.553 (\pm 0.066)$	$0.585 (\pm 0.098$	$0.673 (\pm 0.071)$	$0.673 (\pm 0.071$	$0.547 (\pm 0.057)$	$0.692 (\pm 0.142)$
平均值	$0.711 (\pm 0.071)$	$0.757 (\pm 0.052)$	$0.757 (\pm 0.052)$	$0.751 (\pm 0.049)$	$0.682 (\pm 0.071)$	$0.709 (\pm 0.076)$	$0.735 (\pm 0.063)$	$0.764 (\pm 0.038)$	$0.812 (\pm 0.063)$

Task	Deep-Coral
		Heuristic				Theoretical error guarantees
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.656 (\pm 0.077)$	$0.689 (\pm 0.019)$	$0.689 (\pm 0.019)$	$0.689 (\pm 0.019)$	$0.667 (\pm 0.058)$	$0.656 (\pm 0.077)$	$0.656 (\pm 0.077)$	$0.689 (\pm 0.069)$	$0.711 (\pm 0.051)$
$20 \to 30$	$0.827 (\pm 0.033)$	$0.891 (\pm 0.029)$	$0.891 (\pm 0.029)$	$0.872 (\pm 0.044)$	$0.84 (\pm 0.04)$	$0.827 (\pm 0.033)$	$0.827 (\pm 0.033)$	$0.885 (\pm 0.033)$	$0.904 (\pm 0.038)$
$35 \to 31$	$0.619 (\pm 0.086)$	0.69(±0.071)	$0.69 (\pm 0.071)$	$0.667 (\pm 0.082)$	$0.532 (\pm 0.122)$	$0.619 (\pm 0.086)$	$0.619 (\pm 0.086)$	$0.69 (\pm 0.048)$	$0.706 (\pm 0.06)$
$6 \to 19$	$0.722 (\pm 0.044)$	$0.732 (\pm 0.053)$	$0.732 (\pm 0.053)$	$0.707 (\pm 0.035)$	$0.717 (\pm 0.068)$	$0.722 (\pm 0.044)$	$0.722 (\pm 0.044)$	$0.722 (\pm 0.009$	$0.788 (\pm 0.139)$
$7 \to 18$	$0.648 (\pm 0.047)$	$0.61 (\pm 0.054)$	$0.61 (\pm 0.054)$	$0.629 (\pm 0.071)$	$0.566 (\pm 0.082)$	$0.648 (\pm 0.047$	$0.528 (\pm 0.218)$	$0.629 (\pm 0.039)$	$0.654 (\pm 0.029)$
Avg.	$0.694 (\pm 0.057)$	$0.723 (\pm 0.046)$	$0.723 (\pm 0.046)$	$0.713 (\pm 0.051)$	$0.664 (\pm 0.074)$	$0.694 (\pm 0.057)$	$0.67 (\pm 0.091)$	$0.723 (\pm 0.04)$	$0.753 (\pm 0.063)$

任务	深度珊瑚
		启发式				理论误差保证
	SO	TMV	TMR	TCR	SOR	IWV	DEV	IWA (我们的)	TB
$18 \to 23$	$0.656 (\pm 0.077)$	$0.689 (\pm 0.019)$	$0.689 (\pm 0.019)$	$0.689 (\pm 0.019)$	$0.667 (\pm 0.058)$	$0.656 (\pm 0.077)$	$0.656 (\pm 0.077)$	$0.689 (\pm 0.069)$	$0.711 (\pm 0.051)$
$20 \to 30$	$0.827 (\pm 0.033)$	$0.891 (\pm 0.029)$	$0.891 (\pm 0.029)$	$0.872 (\pm 0.044)$	$0.84 (\pm 0.04)$	$0.827 (\pm 0.033)$	$0.827 (\pm 0.033)$	$0.885 (\pm 0.033)$	$0.904 (\pm 0.038)$
$35 \to 31$	$0.619 (\pm 0.086)$	0.69(±0.071)	$0.69 (\pm 0.071)$	$0.667 (\pm 0.082)$	$0.532 (\pm 0.122)$	$0.619 (\pm 0.086)$	$0.619 (\pm 0.086)$	$0.69 (\pm 0.048)$	$0.706 (\pm 0.06)$
$6 \to 19$	$0.722 (\pm 0.044)$	$0.732 (\pm 0.053)$	$0.732 (\pm 0.053)$	$0.707 (\pm 0.035)$	$0.717 (\pm 0.068)$	$0.722 (\pm 0.044)$	$0.722 (\pm 0.044)$	$0.722 (\pm 0.009$	$0.788 (\pm 0.139)$
$7 \to 18$	$0.648 (\pm 0.047)$	$0.61 (\pm 0.054)$	$0.61 (\pm 0.054)$	$0.629 (\pm 0.071)$	$0.566 (\pm 0.082)$	$0.648 (\pm 0.047$	$0.528 (\pm 0.218)$	$0.629 (\pm 0.039)$	$0.654 (\pm 0.029)$
平均值	$0.694 (\pm 0.057)$	$0.723 (\pm 0.046)$	$0.723 (\pm 0.046)$	$0.713 (\pm 0.051)$	$0.664 (\pm 0.074)$	$0.694 (\pm 0.057)$	$0.67 (\pm 0.091)$	$0.723 (\pm 0.04)$	$0.753 (\pm 0.063)$

Task	CDAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.767 (\pm 0.033)$	$0.678 (\pm 0.038)$	$0.678 (\pm 0.038)$	$0.722 (\pm 0.038)$	$0.756 (\pm 0.019)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	0.7(±0.0)	$0.767 (\pm 0.033)$
$20 \to 30$	$0.872 (\pm 0.029)$	$0.853 (\pm 0.029)$	$0.853 (\pm 0.029)$	$0.84 (\pm 0.011)$	$0.872 (\pm 0.04)$	$0.872 (\pm 0.029$	$0.833 (\pm 0.04)$	$0.84 (\pm 0.029)$	$0.878 (\pm 0.044)$
$35 \to 31$	$0.563 (\pm 0.05)$	$0.738 (\pm 0.041)$	$0.738 (\pm 0.041)$	$0.746 (\pm 0.036)$	$0.651 (\pm 0.144)$	$0.563 (\pm 0.05)$	$0.722 (\pm 0.172)$	$0.722 (\pm 0.014)$	$0.77 (\pm 0.06)$
$6 \to 19$	$0.869 (\pm 0.044)$	$0.833 (\pm 0.052)$	$0.833 (\pm 0.052)$	$0.838 (\pm 0.049)$	$0.838 (\pm 0.049)$	$0.869 (\pm 0.044)$	$0.717 (\pm 0.306)$	$0.848 (\pm 0.045)$	$0.869 (\pm 0.044)$
$7 \to 18$	$0.73 (\pm 0.093)$	$0.711 (\pm 0.115)$	$0.711 (\pm 0.115)$	$0.761 (\pm 0.039)$	$0.723 (\pm 0.058)$	$0.73 (\pm 0.093)$	$0.711 (\pm 0.126)$	$0.717 (\pm 0.068)$	$0.767 (\pm 0.076)$
Avg.	$0.76 (\pm 0.05)$	$0.762 (\pm 0.055)$	$0.762 (\pm 0.055)$	$0.781 (\pm 0.035)$	$0.768 (\pm 0.062)$	$0.76 (\pm 0.05)$	$0.75 (\pm 0.135)$	$0.765 (\pm 0.031)$	$0.81 (\pm 0.052)$

任务	CDAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.767 (\pm 0.033)$	$0.678 (\pm 0.038)$	$0.678 (\pm 0.038)$	$0.722 (\pm 0.038)$	$0.756 (\pm 0.019)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	0.7(±0.0)	$0.767 (\pm 0.033)$
$20 \to 30$	$0.872 (\pm 0.029)$	$0.853 (\pm 0.029)$	$0.853 (\pm 0.029)$	$0.84 (\pm 0.011)$	$0.872 (\pm 0.04)$	$0.872 (\pm 0.029$	$0.833 (\pm 0.04)$	$0.84 (\pm 0.029)$	$0.878 (\pm 0.044)$
$35 \to 31$	$0.563 (\pm 0.05)$	$0.738 (\pm 0.041)$	$0.738 (\pm 0.041)$	$0.746 (\pm 0.036)$	$0.651 (\pm 0.144)$	$0.563 (\pm 0.05)$	$0.722 (\pm 0.172)$	$0.722 (\pm 0.014)$	$0.77 (\pm 0.06)$
$6 \to 19$	$0.869 (\pm 0.044)$	$0.833 (\pm 0.052)$	$0.833 (\pm 0.052)$	$0.838 (\pm 0.049)$	$0.838 (\pm 0.049)$	$0.869 (\pm 0.044)$	$0.717 (\pm 0.306)$	$0.848 (\pm 0.045)$	$0.869 (\pm 0.044)$
$7 \to 18$	$0.73 (\pm 0.093)$	$0.711 (\pm 0.115)$	$0.711 (\pm 0.115)$	$0.761 (\pm 0.039)$	$0.723 (\pm 0.058)$	$0.73 (\pm 0.093)$	$0.711 (\pm 0.126)$	$0.717 (\pm 0.068)$	$0.767 (\pm 0.076)$
平均值	$0.76 (\pm 0.05)$	$0.762 (\pm 0.055)$	$0.762 (\pm 0.055)$	$0.781 (\pm 0.035)$	$0.768 (\pm 0.062)$	$0.76 (\pm 0.05)$	$0.75 (\pm 0.135)$	$0.765 (\pm 0.031)$	$0.81 (\pm 0.052)$

Task	DANN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.689 (\pm 0.107)$	$0.778 (\pm 0.069)$	$0.778 (\pm 0.069)$	$0.778 (\pm 0.069)$	$0.756 (\pm 0.038)$	$0.667 (\pm 0.088)$	$0.656 (\pm 0.077)$	$0.767 (\pm 0.0)$	$0.778 (\pm 0.019)$
$20 \to 30$	$0.795 (\pm 0.068)$	$0.846 (\pm 0.0)$	$0.846 (\pm 0.0)$	$0.859 (\pm 0.022)$	0.865(±0.033	$0.795 (\pm 0.068)$	$0.795 (\pm 0.068)$	$0.846 (\pm 0.019)$	$0.885 (\pm 0.051)$
$35 \to 31$	$0.651 (\pm 0.107)$	$0.77 (\pm 0.014)$	$0.77 (\pm 0.014)$	$0.77 (\pm 0.027)$	$0.659 (\pm 0.159)$	$0.651 (\pm 0.107)$	$0.651 (\pm 0.107)$	$0.77 (\pm 0.027)$	$0.746 (\pm 0.027)$
$6 \to 19$	$0.788 (\pm 0.124)$	$0.914 (\pm 0.046)$	$0.914 (\pm 0.046)$	$0.924 (\pm 0.052)$	$0.854 (\pm 0.126)$	$0.788 (\pm 0.124)$	$0.788 (\pm 0.124)$	0.899(±0.061	$0.919 (\pm 0.052)$
$7 \to 18$	$0.698 (\pm 0.033)$	$0.704 (\pm 0.029)$	$0.704 (\pm 0.029)$	$0.648 (\pm 0.029)$	$0.591 (\pm 0.104)$	$0.698 \hat{i} \pm 0.033$	$0.623 \hat{i} \pm 0.191$	$0.61 (\pm 0.054)$	$0.698 (\pm 0.029)$
Avg.	$0.724 (\pm 0.088)$	$0.802 (\pm 0.032)$	$0.802 (\pm 0.032)$	$0.796 (\pm 0.04)$	$0.745 (\pm 0.092)$	$0.72 (\pm 0.084)$	$0.702 (\pm 0.113)$	$0.778 (\pm 0.032)$	$0.805 (\pm 0.036)$

任务	DANN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.689 (\pm 0.107)$	$0.778 (\pm 0.069)$	$0.778 (\pm 0.069)$	$0.778 (\pm 0.069)$	$0.756 (\pm 0.038)$	$0.667 (\pm 0.088)$	$0.656 (\pm 0.077)$	$0.767 (\pm 0.0)$	$0.778 (\pm 0.019)$
$20 \to 30$	$0.795 (\pm 0.068)$	$0.846 (\pm 0.0)$	$0.846 (\pm 0.0)$	$0.859 (\pm 0.022)$	0.865(±0.033	$0.795 (\pm 0.068)$	$0.795 (\pm 0.068)$	$0.846 (\pm 0.019)$	$0.885 (\pm 0.051)$
$35 \to 31$	$0.651 (\pm 0.107)$	$0.77 (\pm 0.014)$	$0.77 (\pm 0.014)$	$0.77 (\pm 0.027)$	$0.659 (\pm 0.159)$	$0.651 (\pm 0.107)$	$0.651 (\pm 0.107)$	$0.77 (\pm 0.027)$	$0.746 (\pm 0.027)$
$6 \to 19$	$0.788 (\pm 0.124)$	$0.914 (\pm 0.046)$	$0.914 (\pm 0.046)$	$0.924 (\pm 0.052)$	$0.854 (\pm 0.126)$	$0.788 (\pm 0.124)$	$0.788 (\pm 0.124)$	0.899(±0.061	$0.919 (\pm 0.052)$
$7 \to 18$	$0.698 (\pm 0.033)$	$0.704 (\pm 0.029)$	$0.704 (\pm 0.029)$	$0.648 (\pm 0.029)$	$0.591 (\pm 0.104)$	$0.698 \hat{i} \pm 0.033$	$0.623 \hat{i} \pm 0.191$	$0.61 (\pm 0.054)$	$0.698 (\pm 0.029)$
平均值	$0.724 (\pm 0.088)$	$0.802 (\pm 0.032)$	$0.802 (\pm 0.032)$	$0.796 (\pm 0.04)$	$0.745 (\pm 0.092)$	$0.72 (\pm 0.084)$	$0.702 (\pm 0.113)$	$0.778 (\pm 0.032)$	$0.805 (\pm 0.036)$

Task	DSAN
		Heuristic				Theoretical error guarantees
	so	TMV	TMR	TCR	SOR	IWV	DEV	IWA (ours)	TB
$18 \to 23$	$0.756 (\pm 0.051)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	$0.778 (\pm 0.038)$	$0.711 (\pm 0.051)$	$0.756 (\pm 0.051)$	$0.756 (\pm 0.051)$	$0.744 (\pm 0.019)$	$0.8 (\pm 0.0)$
$20 \to 30$	$0.872 (\pm 0.011)$	$0.859 (\pm 0.022)$	$0.859 (\pm 0.022)$	$0.846 (\pm 0.0)$	$0.84 (\pm 0.011)$	$0.872 (\pm 0.011)$	$0.731 (\pm 0.233)$	$0.865 (\pm 0.0)$	$0.872 (\pm 0.011)$
$35 \to 31$	$0.571 (\pm 0.024)$	$0.786 (\pm 0.0)$	$0.786 (\pm 0.0)$	$0.778 (\pm 0.014)$	$0.698 (\pm 0.151)$	$0.571 (\pm 0.024)$	$0.571 (\pm 0.024)$	$0.778 (\pm 0.014)$	$0.817 (\pm 0.027)$
$6 \to 19$	$0.879 (\pm 0.015)$	$0.864 (\pm 0.052)$	$0.864 (\pm 0.052)$	$0.833 (\pm 0.052)$	$0.889 (\pm 0.009$	$0.879 (\pm 0.015)$	$0.662 (\pm 0.363)$	$0.838 (\pm 0.049)$	$0.879 (\pm 0.009)$
$7 \to 18$	$0.717 (\pm 0.05)$	$0.572 (\pm 0.076)$	$0.572 (\pm 0.076)$	$0.547 (\pm 0.068)$	$0.648 (\pm 0.054$	$0.717 (\pm 0.05)$	0.597(±0.076)	$0.384 (\pm 0.047)$	$0.717 (\pm 0.05)$
Avg.	$0.759 (\pm 0.03)$	$0.769 (\pm 0.037)$	$0.769 (\pm 0.037)$	$0.756 (\pm 0.035)$	$0.757 (\pm 0.055)$	$0.759 (\pm 0.03)$	$0.663 (\pm 0.149)$	$0.722 (\pm 0.026)$	$0.817 (\pm 0.019)$

任务	DSAN
		启发式				理论误差保证
	所以	TMV	TMR	TCR	SOR	IWV	DEV	IWA（我们的）	TB
$18 \to 23$	$0.756 (\pm 0.051)$	$0.767 (\pm 0.033)$	$0.767 (\pm 0.033)$	$0.778 (\pm 0.038)$	$0.711 (\pm 0.051)$	$0.756 (\pm 0.051)$	$0.756 (\pm 0.051)$	$0.744 (\pm 0.019)$	$0.8 (\pm 0.0)$
$20 \to 30$	$0.872 (\pm 0.011)$	$0.859 (\pm 0.022)$	$0.859 (\pm 0.022)$	$0.846 (\pm 0.0)$	$0.84 (\pm 0.011)$	$0.872 (\pm 0.011)$	$0.731 (\pm 0.233)$	$0.865 (\pm 0.0)$	$0.872 (\pm 0.011)$
$35 \to 31$	$0.571 (\pm 0.024)$	$0.786 (\pm 0.0)$	$0.786 (\pm 0.0)$	$0.778 (\pm 0.014)$	$0.698 (\pm 0.151)$	$0.571 (\pm 0.024)$	$0.571 (\pm 0.024)$	$0.778 (\pm 0.014)$	$0.817 (\pm 0.027)$
$6 \to 19$	$0.879 (\pm 0.015)$	$0.864 (\pm 0.052)$	$0.864 (\pm 0.052)$	$0.833 (\pm 0.052)$	$0.889 (\pm 0.009$	$0.879 (\pm 0.015)$	$0.662 (\pm 0.363)$	$0.838 (\pm 0.049)$	$0.879 (\pm 0.009)$
$7 \to 18$	$0.717 (\pm 0.05)$	$0.572 (\pm 0.076)$	$0.572 (\pm 0.076)$	$0.547 (\pm 0.068)$	$0.648 (\pm 0.054$	$0.717 (\pm 0.05)$	0.597(±0.076)	$0.384 (\pm 0.047)$	$0.717 (\pm 0.05)$
平均值	$0.759 (\pm 0.03)$	$0.769 (\pm 0.037)$	$0.769 (\pm 0.037)$	$0.756 (\pm 0.035)$	$0.757 (\pm 0.055)$	$0.759 (\pm 0.03)$	$0.663 (\pm 0.149)$	$0.722 (\pm 0.026)$	$0.817 (\pm 0.019)$

The recent surge in generative AI, particularly involving large language models (LLMs), has demonstrated their wide-ranging applicability across various domains (Badita, 2022; Degrave, 2022). These models have enhanced the functionality of tools for search-based interactions (YouWrite, 2022; Writesonic, 2022; Microsoft, 2023), program synthesis (Jain et al., 2021; Romera-Paredes et al., 2023; Key et al., 2023), chat-based interactions (ReplikaAI, 2016; OpenAI, 2022; Google, 2023), and many more. Moreover, language-based approaches have facilitated connections between different modalities, enabling text-to-image (Ramesh et al., 2021; Saharia et al., 2022), text-to-video (Singer et al., 2022), text-to-3D (Poole et al., 2022), text-to-audio (Oord et al., 2016; Wang et al., 2017), and text-to-code (Wang et al., 2021b; Lu et al., 2021; Li et al., 2022b) transformations, to name a few. Consequently, by training on vast quantities of unlabelled textual data, LLMs have been shown to not only store factual knowledge (Petroni et al., 2019; Kassner et al., 2020) and approximate users' intentions to some extent (Andreas, 2022), but also to unlock deep specialist capabilities through innovative prompting techniques (Nori et al., 2023).

最近生成性人工智能的激增，特别是涉及大型语言模型（LLMs），展示了它们在各个领域的广泛适用性（Badita, 2022; Degrave, 2022）。这些模型增强了基于搜索的交互工具的功能（YouWrite, 2022; Writesonic, 2022; Microsoft, 2023）、程序合成（Jain et al., 2021; Romera-Paredes et al., 2023; Key et al., 2023）、基于聊天的交互（ReplikaAI, 2016; OpenAI, 2022; Google, 2023）等。此外，基于语言的方法促进了不同模态之间的连接，使得文本到图像（Ramesh et al., 2021; Saharia et al., 2022）、文本到视频（Singer et al., 2022）、文本到3D（Poole et al., 2022）、文本到音频（Oord et al., 2016; Wang et al., 2017）和文本到代码（Wang et al., 2021b; Lu et al., 2021; Li et al., 2022b）等转换成为可能。因此，通过在大量未标记的文本数据上进行训练，LLMs不仅被证明能够存储事实知识（Petroni et al., 2019; Kassner et al., 2020）并在一定程度上近似用户的意图（Andreas, 2022），还能够通过创新的提示技术解锁深层专业能力（Nori et al., 2023）。

Despite their versatility, current LLMs face challenges such as fallacious reasoning and the generation of erroneous content, commonly referred to as hallucinations (Jones & Steinhardt, 2022). These limitations highlight the importance of integrating complementary symbolic methods to validate and guide the generative processes of LLMs, ensuring more accurate and reliable outputs. In parallel, efforts have focused on developing tool-based approaches (Schick et al., 2023) or template frameworks (Chase, 2023) to extend LLMs' capabilities and enable a broader spectrum of applications. However, these efforts only partially capture the potential inherent in leveraging LLMs as semantic parsers. In contrast to parsers for structured languages a semantic parser is able to break down unstructured human language into semantically meaningful components and transform those into a structured form. While traditionally semantic parsing has been a role filled by specialized algorithms and models, we posit that LLMs, through their training language processing capabilities. In turn, we identify LLMs as a central component in creating sophisticated neuro-symbolic (NeSy) AI systems. These systems integrate symbolic and sub-symbolic concepts and utilize the capabilities of semantic parsing to develop symbolic expressions that enable new probabilistic programming paradigms.

尽管当前的大型语言模型（LLMs）具有多功能性，但仍面临诸如谬误推理和生成错误内容（通常称为幻觉）（Jones & Steinhardt, 2022）等挑战。这些局限性突显了整合互补符号方法的重要性，以验证和指导LLMs的生成过程，从而确保输出更准确和可靠。与此同时，研究者们也专注于开发基于工具的方法（Schick et al., 2023）或模板框架（Chase, 2023），以扩展LLMs的能力并使其能够应用于更广泛的领域。然而，这些努力仅部分捕捉了利用LLMs作为语义解析器所固有的潜力。与结构化语言的解析器不同，语义解析器能够将非结构化的人类语言分解为具有语义意义的组成部分，并将其转换为结构化形式。虽然传统上语义解析是由专门的算法和模型完成的，但我们认为LLMs通过其语言处理能力可以承担这一角色。因此，我们将LLMs视为创建复杂神经符号（NeSy）人工智能系统的核心组成部分。这些系统整合了符号和亚符号概念，并利用语义解析的能力来开发符号表达，从而启用新的概率编程范式。

We introduce SymbolicAI, a compositional NeSy framework able to represent and manipulate multi-modal and self-referential structures (Schmidhuber, 2007; Fernando et al., 2023). SymbolicAI augments the generative process of LLMs with in-context learning operations, realized through functional primitives, and enables the creation of versatile applications through in-context learning (Wei et al., 2022a). These operations enable logic-based components that guide the generative process and enable a modular NeSy system, including a wide range of existing solvers, formal language engines for mathematical expression evaluation, theorem provers, knowledge bases, and search engines for information retrieval. SymbolicAI exposes these solvers as building blocks for constructing compositional functions as computational graphs, making it possible to bridge classical and differentiable programming paradigms with the aim to create domain-invariant problem solvers. In designing the architecture of SymbolicAI, we drew inspiration from a body of evidence that suggests the human brain possesses a selective language processing module (Macsweeney, 2002; Fedorenko et al., 2010; Menenti et al., 2011; Regev et al., 2013; Scott et al., 2016; Deniz et al., 2019; Hu et al., 2022), prior research on cognitive architectures (Newell & Simon, 1956; Newell et al., 1957; Newell & Simon, 1972; Newell, 1990; Laird, 2022), and the significance of language on the structure of semantic maps in the human brain (Huth et al., 2016). We consider language as a central processing module, distinct from other cognitive processes such as reasoning or memory (Paischer et al., 2022; 2023). We hypothesize that such a central processing module based in language is a core component of broad AI systems (see Appendix Section A) and enables the development of fully autonomous AI systems for decision-making.

我们介绍了 SymbolicAI，这是一个组合式的 NeSy 框架，能够表示和操作多模态和自指结构（Schmidhuber, 2007; Fernando et al., 2023）。SymbolicAI 通过功能原语增强了 LLM 的生成过程，结合了上下文学习操作，并通过上下文学习实现了多功能应用的创建（Wei et al., 2022a）。这些操作使得基于逻辑的组件能够引导生成过程，并实现一个模块化的 NeSy 系统，包括广泛的现有求解器、用于数学表达评估的形式语言引擎、定理证明器、知识库和信息检索的搜索引擎。SymbolicAI 将这些求解器作为构建组合函数的构件暴露为计算图，使得能够桥接经典和可微编程范式，旨在创建领域不变的问题求解器。在设计 SymbolicAI 的架构时，我们受到了一系列证据的启发，这些证据表明人类大脑具有选择性语言处理模块（Macsweeney, 2002; Fedorenko et al., 2010; Menenti et al., 2011; Regev et al., 2013; Scott et al., 2016; Deniz et al., 2019; Hu et al., 2022），以及对认知架构的先前研究（Newell & Simon, 1956; Newell et al., 1957; Newell & Simon, 1972; Newell, 1990; Laird, 2022），以及语言在大脑语义图结构中的重要性（Huth et al., 2016）。我们将语言视为一个核心处理模块，与推理或记忆等其他认知过程有所区别（Paischer et al., 2022; 2023）。我们假设这样一个基于语言的核心处理模块是广泛 AI 系统的核心组成部分（见附录 A 节），并使得完全自主的 AI 系统在决策制定中得以发展。

A significant challenge encountered in the development of our framework pertained to the evaluation of LLMs when used as semantic parsers in a NeSy workflow. Current evaluation of generated content relies on metrics for single-step generative processes, such as the BLEU score (Papineni et al., 2002). These metrics are not suitable for evaluating multi-step generative processes. BLEU has limitations,as it measures

n

-gram-based overlap of generated output with a reference that does not consider the semantic meaning. As a result, BLEU fails to capture semantic equivalence, especially in more complex tasks. More recent metrics such as CIDEr (Vedantam et al., 2014) or SPICE (Anderson et al., 2016) are also not suitable for our problem, either because they are built upon BLEU (in case of CIDEr) or designed with inductive biases specifically for image captioning.

在我们框架的开发中遇到的一个重大挑战与在 NeSy 工作流程中将 LLM 作为语义解析器时的评估有关。目前对生成内容的评估依赖于单步生成过程的度量标准，例如 BLEU 分数（Papineni 等，2002）。这些度量标准不适合评估多步生成过程。BLEU 有其局限性，因为它测量生成输出与参考之间的

n

-gram 基重叠，而不考虑语义意义。因此，BLEU 无法捕捉语义等价性，尤其是在更复杂的任务中。更近期的度量标准，如 CIDEr（Vedantam 等，2014）或 SPICE（Anderson 等，2016），也不适合我们的问题，原因要么是它们基于 BLEU（在 CIDEr 的情况下），要么是专门为图像描述设计的归纳偏见。

Therefore, alongside our framework we introduce a quality measure (and its empirical score) for multi-step generative processes based on semantic meaning. We refer to our score as "Vector Embedding for Relational Trajectory Evaluation through Cross-similarity", or VERTEX score for short. Our VERTEX score uses embeddings to compare node distributions within a computational graph. It measures the semantic meaning across the distributional path by computing at each node the cross-similarity between the generated embeddings and embeddings sampled from a reference

因此，除了我们的框架外，我们还引入了一种基于语义意义的多步生成过程的质量度量（及其经验分数）。我们将我们的分数称为“通过交叉相似性进行关系轨迹评估的向量嵌入”，简称 VERTEX 分数。我们的 VERTEX 分数使用嵌入来比较计算图中的节点分布。它通过计算每个节点生成的嵌入与从参考分布中采样的嵌入之间的交叉相似性，来测量分布路径上的语义意义。

distribution. Furthermore, the VERTEX score is designed such that it can be used as a reward signal in a reinforcement learning setting (Sutton, 1984). Finally, we propose a benchmark for evaluating complex workflows. We define a set of basic evaluations, particularly associative predictions based on in-context learning, multi-modal bindings for tool utilization, and program synthesis for subroutine execution. Furthermore, we introduce complex evaluations for logic-based components and hierarchical computational graphs.

此外，VERTEX 分数的设计使其可以作为强化学习环境中的奖励信号（Sutton，1984）。最后，我们提出了一个用于评估复杂工作流程的基准。我们定义了一组基本评估，特别是基于上下文学习的联想预测、工具利用的多模态绑定以及子例程执行的程序合成。此外，我们还引入了针对基于逻辑的组件和分层计算图的复杂评估。

Symbolic Methods The field of symbolic AI has its foundations in the works of the Logic Theorist (LT) (Newell & Simon, 1956) and the General Problem Solver (GPS) (Newell et al., 1957). These programs represented the first steps towards automated reasoning and problem-solving utilizing symbolic representations. Despite their advancements, both faced challenges in dealing with the complexity of real-world problems, particularly due to the combinatorial nature of the solution space. To address these limitations, the Soar (Laird et al., 1987) cognitive architecture was developed, advancing the notion that intelligent behavior results from goal-oriented search through a problem space (Newell & Simon, 1972; McCarthy et al., 2006), with each step consisting of selecting and applying operators. Soar introduced components like reinforcement learning, impasses, sub-states, and chunking to enhance its problem-solving capabilities. It also demonstrated the importance of learning from experiences to adapt and improve performance over time. However, Santoro et al. (2022) emphasizes the subjectivity of symbols and suggests that human-like symbolic fluency could develop in machines through learning algorithms immersed in socio-cultural contexts. This perspective, anchored in the notion that symbols are triadic and their meaning emerges from consensus, seeks to move away from traditional symbolic AI methodologies towards AI that adaptively learns meaning and behaviors from human-like experiences. The goal is to cultivate machines that demonstrate symbolic behaviors across a spectrum of competencies, potentially mirroring the evolutionary and social learning processes observed in humans. Lastly, symbolic AI struggles with real-world data's unpredictability and variability. These challenges have led to the employment of statistical learning methodologies, like deep learning (Alom et al., 2018), which are more adept at managing noise and uncertain information through vector-valued representations.

符号方法符号人工智能领域的基础源于逻辑理论家（LT）（Newell & Simon, 1956）和通用问题求解器（GPS）（Newell et al., 1957）的研究。这些程序代表了利用符号表示进行自动推理和问题解决的第一步。尽管取得了一定进展，但两者在处理现实世界问题的复杂性方面面临挑战，特别是由于解决方案空间的组合性质。为了解决这些局限性，Soar（Laird et al., 1987）认知架构的开发推进了智能行为源于通过问题空间的目标导向搜索的概念（Newell & Simon, 1972; McCarthy et al., 2006），每一步都包括选择和应用操作符。Soar引入了强化学习、僵局、子状态和块化等组件，以增强其问题解决能力。它还展示了从经验中学习的重要性，以便随着时间的推移适应和提高性能。然而，Santoro等（2022）强调了符号的主观性，并建议通过沉浸于社会文化背景中的学习算法，机器可能会发展出类人符号流畅性。这种观点基于符号是三元的，其意义源于共识，旨在摆脱传统符号人工智能方法，朝着能够适应性地从类人经验中学习意义和行为的人工智能发展。目标是培养能够在多种能力范围内展示符号行为的机器，可能反映出人类观察到的进化和社会学习过程。最后，符号人工智能在应对现实世界数据的不可预测性和变异性方面存在困难。这些挑战促使采用统计学习方法，如深度学习（Alom et al., 2018），这些方法更擅长通过向量值表示来处理噪声和不确定信息。

Sub-Symbolic Methods The sub-symbolic framework, rooted in neural network paradigms, began with pioneering works such as the perceptron (McCulloch & Pitts, 1943), with the first hardware implementation quickly following (Rosenblatt, 1958). The foundational notion of distributed processing (Rumelhart et al., 1986) was later bolstered and further expanded by demonstrating that multilayer feedforward networks with a single hidden layer can serve as universal approximators for any Borel measurable function, given sufficient hidden units (Hornik et al., 1989). Fast-forward, contemporary frameworks achieve a significant leap with the introduction of the Transformer architecture (Vaswani et al., 2017), which underpins most of today's LLMs. These LLMs demonstrate exceptional capabilities in in-context learning, a method popularized by the likes of GPT-3 (Brown et al., 2020), where models improve task performance through natural language instruction and examples provided directly in the input prompt. While in-context learning bypasses the need for explicit retraining, it demands meticulous prompt design to steer models towards desired behaviors.

子符号方法子符号框架根植于神经网络范式，始于感知器等开创性工作（McCulloch & Pitts, 1943），随后迅速出现了首个硬件实现（Rosenblatt, 1958）。分布式处理的基础概念（Rumelhart et al., 1986）后来得到了加强，并进一步扩展，证明了具有单个隐藏层的多层前馈网络可以作为任何博雷尔可测函数的通用逼近器，只要隐藏单元足够（Hornik et al., 1989）。快进到现代框架，引入了Transformer架构（Vaswani et al., 2017），这为当今大多数LLM提供了基础。这些LLM在上下文学习方面表现出色，这一方法由GPT-3等模型推广（Brown et al., 2020），模型通过自然语言指令和直接在输入提示中提供的示例来提高任务性能。虽然上下文学习绕过了显式再训练的需要，但它要求精心设计提示，以引导模型朝向期望的行为。

Neuro-Symbolic Methods To overcome the limitations of each individual method, NeSy approaches meld the statistical inference strengths of deep neural architectures with the generalization and explainability of symbolic systems (Garcez et al., 2015; Besold et al., 2017; d'Avila Garcez et al., 2019; d'Avila Garcez & Lamb, 2020; Lamb et al., 2020; Hamilton et al., 2022; Yu et al., 2023). Some approaches focus on different strategies for integrating learning and reasoning processes (Yu et al., 2023; Fang et al., 2024). Firstly, learning for reasoning methods treat the learning aspect as an accelerator for reasoning, in which deep neural networks are employed to reduce the search space for symbolic systems (Silver et al., 2016; 2017b;a; Qu & Tang, 2019; Schrittwieser et al., 2020). Secondly, reasoning for learning views reasoning as a way to regularize learning, in which symbolic knowledge acts as a guiding constraint that a symbiotic relationship between learning and reasoning. Here, both elements interact and share information to boost problem-solving capabilities (Donadello et al., 2017; Manhaeve et al., 2018; Mao et al., 2019; Ellis, 2023). This synergy further extends when considering graph-based methods, which closely align with the objectives of our proposed framework. Research in this area, such as CycleGT (Guo et al., 2020) and Paper2vec (Ganguly & Pudi, 2017) explores unsupervised techniques for bridging graph and text representations, GPTSwarm (Zhuge et al., 2024) explores graph optimizers to refine node-level prompts and edge optimization. Subsequently, graph embeddings, when utilized within symbolic frameworks, can enhance knowledge graph reasoning tasks (Zhang et al., 2021), or more generally, provide the bedrock for learning domain-invariant representations (Park et al., 2023).

神经符号方法为克服各个单一方法的局限性，NeSy 方法将深度神经架构的统计推理优势与符号系统的泛化能力和可解释性结合在一起（Garcez et al., 2015; Besold et al., 2017; d'Avila Garcez et al., 2019; d'Avila Garcez & Lamb, 2020; Lamb et al., 2020; Hamilton et al., 2022; Yu et al., 2023）。一些方法专注于整合学习和推理过程的不同策略（Yu et al., 2023; Fang et al., 2024）。首先，推理学习方法将学习视为推理的加速器，其中使用深度神经网络来减少符号系统的搜索空间（Silver et al., 2016; 2017b;a; Qu & Tang, 2019; Schrittwieser et al., 2020）。其次，学习推理方法将推理视为一种规范学习的方式，其中符号知识作为指导约束，形成学习与推理之间的共生关系。在这里，这两个元素相互作用并共享信息，以增强问题解决能力（Donadello et al., 2017; Manhaeve et al., 2018; Mao et al., 2019; Ellis, 2023）。当考虑基于图的方法时，这种协同作用进一步扩展，与我们提出的框架的目标紧密对齐。该领域的研究，如 CycleGT（Guo et al., 2020）和 Paper2vec（Ganguly & Pudi, 2017），探索了弥合图和文本表示的无监督技术，GPTSwarm（Zhuge et al., 2024）探索了图优化器以优化节点级提示和边优化。随后，当图嵌入在符号框架中使用时，可以增强知识图推理任务（Zhang et al., 2021），或更一般地，为学习领域不变表示提供基础（Park et al., 2023）。

Lastly, building upon the insights from Sun et al. (2022), the integration of NeSy techniques in scientific workflows promises significant acceleration in scientific discovery. While previous work has effectively identified opportunities and challenges, we have taken a more ambitious approach by developing a comprehensive framework from the ground up to facilitate a wide range of NeSy integrations.

最后，基于 Sun 等人 (2022) 的见解，将 NeSy 技术整合到科学工作流程中有望显著加速科学发现。尽管之前的工作有效地识别了机会和挑战，我们采取了更为雄心勃勃的方法，从零开始开发一个综合框架，以促进广泛的 NeSy 整合。

Large Language Models In part, instruction-based fine-tuning of LLMs through reinforcement learning from human feedback (Ouyang et al., 2022; Li et al., 2023) or direct preference optimization (Rafailov et al., 2023) has shown promising results dealing with value misalignment issues (Bradley Knox & Stone, 2008; MacGlashan et al., 2017; Christiano et al., 2017; Ibarz et al., 2018; Goyal et al., 2022), unlocking new possibilities for chain of thoughts (Wei recent research also highlights the limitations of LLMs in functional linguistic competence despite their proficiency in formal linguistic competence (Mahowald et al., 2023). Whereas formal linguistic competence encompasses the ability to understand and generate language, functional linguistic competence pertains to the application of language in real-world contexts, such as conveying sensory input or recalling information from memory. Examples of functional linguistic competence include implicatures (Ruis et al., 2022) and contextual language comprehension beyond the statistical manifestation of data distributions (Bransford & Johnson, 1972; Mikolov et al., 2013b). Consequently, operating LLMs through a purely inference-based approach confines their capabilities within their provided context window, severely limiting their horizon. This results in deficiencies for situational modeling, non-adaptability through text length may not yield greater capabilities, as demonstrated by the observed U-shaped performance curve (Liu et al., 2023) where LLMs excel when utilizing information at the beginning or end of the input context, but struggle with information located in the middle, especially as context increases. These challenges are actively being researched, with novel approaches such as Hyena (Poli et al., 2023), RWKV (Bo, 2021), GateLoop (Katsch, 2023), Mamba (Gu & Dao, 2023) and xLSTM (Beck et al., 2024) surfacing. Meanwhile, the re-emergence of interest in retrieval-augmented generative approaches (Li et al., 2022a) offers an alternative by circumventing the autoregressive nature of the widely-utilized Transformer architecture (Vaswani et al., 2017), enabling context enrichment with lateral information.

大型语言模型在一定程度上，通过人类反馈的强化学习（Ouyang et al., 2022; Li et al., 2023）或直接偏好优化（Rafailov et al., 2023）对LLM进行基于指令的微调，已在解决价值不一致问题上显示出良好的效果（Bradley Knox & Stone, 2008; MacGlashan et al., 2017; Christiano et al., 2017; Ibarz et al., 2018; Goyal et al., 2022），为思维链的探索开辟了新的可能性（Wei）。最近的研究还强调了LLM在功能语言能力方面的局限性，尽管它们在形式语言能力方面表现出色（Mahowald et al., 2023）。形式语言能力包括理解和生成语言的能力，而功能语言能力则涉及在现实世界情境中应用语言的能力，例如传达感官输入或从记忆中回忆信息。功能语言能力的例子包括隐含意义（Ruis et al., 2022）和超越数据分布统计表现的上下文语言理解（Bransford & Johnson, 1972; Mikolov et al., 2013b）。因此，通过纯粹的推理基础方法操作LLM会限制其能力在提供的上下文窗口内，严重限制了其视野。这导致了情境建模的不足，文本长度的非适应性可能不会带来更大的能力，正如观察到的U型性能曲线所示（Liu et al., 2023），LLM在利用输入上下文开头或结尾的信息时表现出色，但在处理位于中间的信息时却表现不佳，尤其是在上下文增加时。这些挑战正在积极研究中，出现了新的方法，如Hyena（Poli et al., 2023）、RWKV（Bo, 2021）、GateLoop（Katsch, 2023）、Mamba（Gu & Dao, 2023）和xLSTM（Beck et al., 2024）。与此同时，检索增强生成方法的重新兴起（Li et al., 2022a）提供了一种替代方案，通过规避广泛使用的Transformer架构的自回归特性（Vaswani et al., 2017），使上下文能够与横向信息相结合。

In-Context Learning Recently, several in-context learning methodologies evolved to enable tool usage through LLMs (Schick et al., 2023), or refine the generative outcome of LLMs (Yang et al., 2023). This includes chain-of-thought (CoT) prompting, a method that conditions the model to reveal its step-by-step reasoning process (Wei et al., 2022b; Singhal et al., 2023). CoT prompting breaks down complex tasks into simpler, sequential steps, and helps with interpreting LLM's output. Self-generated CoT, where models are encouraged to generate their own reasoning chains based on training examples, surpasses even expertly crafted CoT (Fernando et al., 2023). This observation echoes other reports that GPT-4 has an emergent self-improving capability through introspection, such as self-verification (Weng et al., 2023) or self-consistency (Wang et al., 2023b). Tree of Thoughts (ToT) enables LLMs to solve complex problems by exploring multiple reasoning paths through a search tree of coherent text units, demonstrating significant problem-solving enhancements in tasks requiring strategic planning and search (Yao et al., 2023a). Ensemble techniques further enhance the robustness and accuracy of model predictions by combining several strategies to establish a consensus (Nori et al., 2023).

在上下文学习中，最近出现了几种上下文学习方法，以通过大型语言模型（LLMs）实现工具使用（Schick et al., 2023），或优化LLMs的生成结果（Yang et al., 2023）。这包括链式思维（CoT）提示，这是一种使模型展示其逐步推理过程的方法（Wei et al., 2022b；Singhal et al., 2023）。CoT提示将复杂任务分解为更简单的顺序步骤，并有助于解释LLM的输出。自生成的CoT，鼓励模型基于训练示例生成自己的推理链，甚至超过了专家精心设计的CoT（Fernando et al., 2023）。这一观察与其他报告相呼应，表明GPT-4具有通过内省自我改善的能力，例如自我验证（Weng et al., 2023）或自我一致性（Wang et al., 2023b）。思维树（ToT）使LLMs能够通过探索多个推理路径来解决复杂问题，利用一致文本单元的搜索树，在需要战略规划和搜索的任务中显示出显著的解决问题能力提升（Yao et al., 2023a）。集成技术通过结合多种策略建立共识，进一步增强了模型预测的稳健性和准确性（Nori et al., 2023）。

Conventional approaches employing foundation models, such as LLMs, are predominantly confined to single-step or few-step executions and primarily reliant on hand-crafted prompt instructions, often referred to as in-context learning. This restricted scope limits the utilization of different modalities, lacks verification, and exhibits limited tool proficiency. We posit that the use of NeSy engines as core computation units, realized through logic-based methodologies coupled with sub-symbolic foundation models, offers a more general, robust, and verifiable perspective. This approach has several advantages. Firstly, it enables the integration of pre-existing solutions (e.g. various classical algorithms), to focus on decision-making (e.g. selecting the respective tool based on in-context classification). Thirdly, it provides an interpretable language-based control layer for explainable, autonomous systems. In the following section, we elaborate on the key design principles underlying SymbolicAI and how we guide the generative processes of NeSy engines. For further technical details, see Appendix Section 5.

传统方法采用基础模型，如大型语言模型（LLMs），主要局限于单步或少步执行，并主要依赖于手工制作的提示指令，通常称为上下文学习。这种受限的范围限制了不同模态的利用，缺乏验证，并表现出有限的工具熟练度。我们认为，使用NeSy引擎作为核心计算单元，通过基于逻辑的方法与亚符号基础模型相结合，提供了一种更通用、强大且可验证的视角。这种方法有几个优点。首先，它能够整合现有的解决方案（例如各种经典算法），以专注于决策制定（例如，根据上下文分类选择相应的工具）。第三，它为可解释的自主系统提供了一个可解释的基于语言的控制层。在接下来的部分中，我们详细阐述了SymbolicAI的关键设计原则，以及我们如何指导NeSy引擎的生成过程。有关更多技术细节，请参见附录第5节。

Symbols and Expressions As posited by Newell & Simon (1976), symbols are elemental carriers of meaning within a computational context

^{3}

. These symbols define physical patterns capable of composing complex structures,and are central to the design and interpretation of logic and knowledge representations (Augusto, 2022). We define a symbol as the set

S = ⋃_{n \geq 0} L^{n}

formed by concatenating characters from a finite character set

L

,i.e. the vocabulary in an LLM setting,and with

n

representing the sequence length of the string. Thus,let the set of all possible symbols be number of symbols from

\sum

,and when evaluated returns a new symbol in

\sum

. For any subset

{S_{1}, S_{2}, \dots, S_{m}} \subseteq \sum

, an expression is defined as

ω : ⨁_{i = 1}^{m} S_{i} \to S^{'}

from the set of all possible expressions

ω \in Ω

,where

S^{'} \in \sum

,and

⨁

represents the placeholder operation of combining and transforming the symbols according to specific rules for

m

number of symbols. Such a specific rule for

⨁

can define an arithmetic expression

⨁ := +

where two symbols are added,i.e.

ω :=

" 1 " + "two" which results in a new symbol " 3 " or "three". Thus,SymbolicAI is based on the concept that symbols, and the expressions they form, are reflections of the information inherent in a NeSy system, and serve as surrogate for the interaction between the NeSy system and the problem space. Moreover, we argue that real patterns (Dennett, 1991), recurring and identifiable structures that coherently and reliably emerge in the data beyond mere randomness or noise, can be effectively realized through symbols.

符号与表达式正如 Newell 和 Simon (1976) 所提出的，符号是在计算环境中承载意义的基本载体

^{3}

。这些符号定义了能够组成复杂结构的物理模式，并且在逻辑和知识表示的设计与解释中占据核心地位 (Augusto, 2022)。我们将符号定义为由有限字符集

L

中的字符连接而成的集合

S = ⋃_{n \geq 0} L^{n}

，即 LLM 环境中的词汇，并且

n

表示字符串的序列长度。因此，所有可能符号的集合可以表示为来自

\sum

的符号数量，当被评估时返回

\sum

中的新符号。对于任何子集

{S_{1}, S_{2}, \dots, S_{m}} \subseteq \sum

，表达式被定义为来自所有可能表达式集合

ω \in Ω

的

ω : ⨁_{i = 1}^{m} S_{i} \to S^{'}

，其中

S^{'} \in \sum

，而

⨁

表示根据特定规则组合和转换符号的占位符操作，适用于

m

数量的符号。对于

⨁

的这种特定规则可以定义一个算术表达式

⨁ := +

，其中两个符号相加，即

ω :=

" 1 " + "two"，结果是一个新符号 " 3 " 或 "three"。因此，SymbolicAI 基于符号及其形成的表达式反映了 NeSy 系统中固有信息的概念，并作为 NeSy 系统与问题空间之间交互的替代物。此外，我们认为真实模式 (Dennett, 1991)，即在数据中超越单纯随机性或噪声而一致可靠地出现的可识别结构，可以通过符号有效实现。

Furthermore, we utilize language as a tool for mapping complex concepts, leveraging its inherent semantics and abstractions to describe states and properties of a problem at hand. These mappings are universal, e.g. they may be utilized to define scene descriptions, long-horizon planning, acoustic properties, emotional states, physical conditions, etc. Therefore, language serves as a comprehensive, yet abstract framework to encapsulate meanings, and refer to it as the convex hull of the knowledge of our society. Subsequently, it is common to attribute existing physical objects with abstract concepts, as exemplified by our natural tendency to link tangible objects to colors and emotions, such as blending the color "red" with "heart", "warm", and "passion". This approach also anchors our work in the field of formal language theory, as we require a structured method to construct mappings from the world to language. Consequently, we use formal language structures, such as grammars, to systematically define our language-centric approach to problem-solving and the associated translation of real-world complexities into linguistic terms.

此外，我们利用语言作为映射复杂概念的工具，利用其固有的语义和抽象来描述当前问题的状态和属性。这些映射是普遍的，例如，它们可以用于定义场景描述、长远规划、声学特性、情感状态、物理条件等。因此，语言作为一个全面而抽象的框架来封装意义，我们将其称为我们社会知识的凸包。随后，常常将现有的物理对象与抽象概念联系起来，例如我们自然倾向于将有形物体与颜色和情感联系在一起，比如将“红色”与“心”、“温暖”和“激情”混合。这种方法也将我们的工作锚定在形式语言理论领域，因为我们需要一种结构化的方法将世界映射到语言。因此，我们使用形式语言结构，如语法，系统地定义我们以语言为中心的问题解决方法，以及将现实世界的复杂性转化为语言术语的相关翻译。

Formal Languages In formal language theory and linguistics, languages are structured following the Chomsky hierarchy, which classifies languages by the complexity of their grammatical structure (Chomsky, 1956). This hierarchy defines four types of grammars (Type-3 to Type-0) and separates formal languages by their grammatical complexity. A grammar in this context consists of terminal and non-terminal symbols, production rules, and a designated start symbol, enabling the generation of valid strings within a language.

形式语言在形式语言理论和语言学中，语言的结构遵循乔姆斯基层次结构，该结构根据语法结构的复杂性对语言进行分类（乔姆斯基，1956）。该层次结构定义了四种类型的语法（类型-3到类型-0），并根据语法复杂性将形式语言分开。在此上下文中，语法由终结符和非终结符符号、产生规则以及指定的起始符号组成，从而能够在语言中生成有效的字符串。

We define a NeSy engine as a mapping

V_{S^{*}} : Ω \times N \times T \to \sum

,where

N \subset \sum

is a set of non-terminal symbols,

T \subset \sum

is a set of terminal symbols and

N \cap T = \emptyset

,and

S^{*} \in \sum

is a starting symbol. We further formalize a grammar

G = (N, T, P, S^{*})

with production rules defined as a

P := V_{S^{*}} (ω, N, T)

. This grammar describes the generation of symbols through expressions

ω

. For simplicity,we will drop the subscript of

V_{S^{*}}

and use it as

V

. We identify LLMs as promising candidates for functioning as part of NeSy engines. In SymbolicAI,a symbol

S

is augmented with conditional instructions and types derived from DSLs, custom defined or not (e.g. HTML, SQL, etc.), tailored for directing the LLMs. The key advantage of LLMs over previous systems lies in their ability to generalize across formal languages (Wang et al., 2023a) and knowledge systems. Although there is currently no universal consensus regarding the precise classification of natural language within the Chomsky hierarchy, our approach can be understood as employing a situation-specific, context-sensitive grammar, which enables the processing of instructions and analogies with a nuanced understanding of language. The intersection between formal and natural languages becomes evident when considering how language patterns, through prompts like "You are a helpful assistant...", elicit structured responses, indicating a potential underlying formal mechanism at play. This observation underlines the utility of such a grammar in our framework, where it serves as an explicit schema guiding the structure of examples for in-context learning. For instance,equating "3.1415..." with "

π

" or "August 4,1961" with "1961-08-04" in a given context demonstrates context-dependent interpretation of symbols. Such a system doesn't rigidly adhere to standard grammatical rules but instead adjusts and interprets based on the context, effectively creating a situation-specific gram-

我们将 NeSy 引擎定义为一个映射

V_{S^{*}} : Ω \times N \times T \to \sum

，其中

N \subset \sum

是一组非终结符，

T \subset \sum

是一组终结符，

N \cap T = \emptyset

和

S^{*} \in \sum

是一个起始符。我们进一步形式化一个文法

G = (N, T, P, S^{*})

，其产生规则定义为

P := V_{S^{*}} (ω, N, T)

。该文法描述了通过表达式

ω

生成符号的过程。为简化起见，我们将省略

V_{S^{*}}

的下标，并将其用作

V

。我们将 LLM 识别为有希望的候选者，能够作为 NeSy 引擎的一部分发挥作用。在 SymbolicAI 中，符号

S

被增强为包含来自 DSL 的条件指令和类型，无论是自定义定义的还是非自定义定义的（例如 HTML、SQL 等），以指导 LLM 的操作。LLM 相较于之前的系统的关键优势在于其能够在形式语言（Wang et al., 2023a）和知识系统之间进行泛化。尽管目前对于自然语言在乔姆斯基层级中的精确分类尚无普遍共识，但我们的方法可以理解为采用一种特定情境的、上下文敏感的文法，这使得能够以细致的语言理解处理指令和类比。当考虑到语言模式如何通过诸如“你是一个有帮助的助手……”这样的提示引发结构化响应时，形式语言和自然语言之间的交集变得显而易见，这表明潜在的形式机制在起作用。这一观察强调了这种文法在我们框架中的实用性，它作为一个明确的模式，指导上下文学习的示例结构。例如，在给定的上下文中，将“3.1415...”等同于“

π

”或将“1961年8月4日”与“1961-08-04”相等，展示了符号的上下文依赖解释。这样的系统并不严格遵循标准语法规则，而是根据上下文进行调整和解释，有效地创建了一种特定情境的文法。

Figure 2: Illustration for NeSy pipeline, showcasing conceptual usage of in-context learning methodologies, domain-specific language (DSL) structures, and the expression evaluations through a NeSy engine based on an LLM and constraint verification. The expression showcases the sorted insert operator

≪

and how the information of the symbol B is included in the symbol AC. The violet placeholder in the DSL Prompt represents an instruction, such as "Insert the right-hand side value into the left-hand value in ascending order:" The positions below represent task-specific few-shot examples. The DSL Prompt receives the expression

ω_{<<}

and maps it to

{\hat{ω}}_{<<}

that can be processed by the LLM-based NeSy function

V_{S^{*}}

and outputs a new symbol.

图 2：NeSy 管道的示意图，展示了上下文学习方法、特定领域语言 (DSL) 结构的概念性使用，以及通过基于 LLM 的 NeSy 引擎进行的表达式评估。该表达式展示了排序插入操作符

≪

以及符号 B 的信息如何包含在符号 AC 中。DSL 提示中的紫色占位符表示一个指令，例如“将右侧值按升序插入到左侧值中：”下面的位置代表特定任务的少量示例。DSL 提示接收表达式

ω_{<<}

并将其映射到可以由基于 LLM 的 NeSy 函数

V_{S^{*}}

处理的

{\hat{ω}}_{<<}

，并输出一个新符号。

Function Composition In SymbolicAI, we use function composition to construct complex hierarchies and behaviors from fundamental elements. Therefore, our framework enables modeling of interconnected processes, where the output of one function is used as input for another, thus creating a sequence of operations. Through function composition, we construct computational graphs, in which intermediate symbols represent the nodes or states within these graphs. Formally,function composition is denoted by

\circ

,where combining functions

f

and

g

yields a new function

h = g \circ f

,defined as

h (x) = g (f (x))

For functions

f : X \to Y

and

g : Y \to Z

,their composition results in a function mapping elements from domain

X

to codomain

Z

through

g (f (x))

. Although traditionally the codomain of any subset relationship between these domains and codomains, which is particularly beneficial for in-context learning. When using LLMs for NeSy production rules

V

,we can derive a multi-step generative process by composing a computational graph as a sequence of zero- and few-shot function compositions:

函数组合在符号人工智能中，我们使用函数组合从基本元素构建复杂的层次结构和行为。因此，我们的框架能够建模相互关联的过程，其中一个函数的输出作为另一个函数的输入，从而创建一系列操作。通过函数组合，我们构建计算图，其中中间符号表示这些图中的节点或状态。形式上，函数组合用

\circ

表示，其中组合函数

f

和

g

产生一个新函数

h = g \circ f

，定义为

h (x) = g (f (x))

。对于函数

f : X \to Y

和

g : Y \to Z

，它们的组合结果是一个将元素从域

X

映射到余域

Z

的函数，通过

g (f (x))

。尽管传统上这些域和余域之间的任何子集关系的余域是有益的，特别是在上下文学习中。当使用大型语言模型进行神经符号生产规则

V

时，我们可以通过将计算图组合为零-shot 和少量-shot 函数组合的序列，推导出一个多步骤生成过程。

where

ω_{0}

is the initial instruction and

j

defines the index variable for a multi-step generative process. By leveraging functional in-context learning, where zero- and few-shot examples act as dynamic elements of the function's domain, SymbolicAI has the ability to interpret and respond to diverse input contexts. For instance, a function can classify a user request and select an appropriate interface (e.g. WolframAlpha) to process the request. The output modality may even vary based on the respective engine. This enables SymbolicAI to handle operations over multi-modal data that connects multi-step generative processes and establishes function composition as a central tenet in bridging multiple modalities and coordinating a variety of tasks.

其中

ω_{0}

是初始指令，而

j

定义了多步骤生成过程的索引变量。通过利用功能性上下文学习，其中零-shot 和少量-shot 示例作为函数域的动态元素，符号人工智能能够解释和响应多样的输入上下文。例如，一个函数可以对用户请求进行分类，并选择一个合适的接口（例如 WolframAlpha）来处理请求。输出模式甚至可能根据各自的引擎而有所不同。这使得符号人工智能能够处理跨多模态数据的操作，连接多步骤生成过程，并将函数组合确立为连接多种模态和协调各种任务的核心原则。

Domain-Invariant Associations In-context learning enabled LLMs to become versatile task solvers by interpolating within the training distribution, to the extent that even potentially unseen tasks are addressable (Brown et al., 2020). We attribute this to associations formed within the input space and the capacity of Transformer architectures for defining domain-invariant feature sub-spaces. This phenomenon has parallels with few-shot learning approaches such as SubGD (Gauch et al., 2022), a method based on identifying and utilizing a low-dimensional subspace, learned from various tasks that effectively regularize the learning process. Since LLMs have been trained on different domains and tasks, which also include formulations of mathematical expressions, we posit that specific tokens, such as the equality sign, can be leveraged to associate meanings between different symbolic objects. Unlike domain-invariant representations that create invariant features across different learning tasks, our approach leverages the in-context generalization capability of LLMs to construct invariant symbolic associations that aim to preserve, manipulate and propagate situational context. We can use these properties to build operations that apply transformations on objects that are substitutes to the semantically aligned few-shot learning examples.

域不变关联在上下文学习的支持下，LLM 能够通过在训练分布内进行插值，成为多功能的任务求解者，以至于即使是潜在未见的任务也可以被处理（Brown et al., 2020）。我们将此归因于在输入空间内形成的关联以及 Transformer 架构定义域不变特征子空间的能力。该现象与少样本学习方法如 SubGD（Gauch et al., 2022）有相似之处，该方法基于识别和利用从各种任务中学习到的低维子空间，有效地规范化学习过程。由于 LLM 已在不同领域和任务上进行训练，其中也包括数学表达式的形式，我们假设特定的标记，例如等号，可以被利用来关联不同符号对象之间的意义。与在不同学习任务中创建不变特征的域不变表示不同，我们的方法利用 LLM 的上下文泛化能力构建不变的符号关联，旨在保持、操作和传播情境上下文。我们可以利用这些属性构建对对象进行变换的操作，这些对象是语义上对齐的少样本学习示例的替代品。

In this section, we discuss the specifics of the proposed SymbolicAI framework. For more details about the framework structure, see Appendix Section C. For installation and usage of our framework, see Appendix Section D. For more technical details and code snippets, see Appendix Section E.

在本节中，我们讨论所提出的 SymbolicAI 框架的具体细节。有关框架结构的更多详细信息，请参见附录 C 节。有关我们框架的安装和使用，请参见附录 D 节。有关更多技术细节和代码片段，请参见附录 E 节。

Types and Representations Analogous to the type object in Python, the base type of SymbolicAI is a symbol represented by the base type Symbol. All other subtypes, such as Expression, represent their mathematical namesake and can be evaluated and simplified. These subtypes inherit from Symbol the base attributes, primitive operators, and helper methods.

类型和表示类似于 Python 中的类型对象，SymbolicAI 的基本类型是由基本类型 Symbol 表示的符号。所有其他子类型，如 Expression，表示其数学名称，并且可以被评估和简化。这些子类型继承了 Symbol 的基本属性、原始操作符和辅助方法。

Although SymbolicAI uses a language-centric design, modeling and manipulating every interaction into symbolic representations is not inherently efficient. Therefore, we establish mappings between symbolic and sub-symbolic representations for sensory inputs and non-discrete elements. Such mappings are typically realized through function approximation. This allows us to map between modality-to-language and language-to-modality use cases. Here, modality serves as a placeholder for various types such as text, image, video, audio, motion, etc. In turn, each Symbol object contains valued and vector-valued representations, obtained through value and embedding attributes. The latter represents a symbol's current value, akin to embedding text and storing it as a PyTorch tensor (Paszke et al., 2019) or NumPy array (Harris et al., 2020). While for an LLM, the numerical tensors may lack inherent meaning, vector-valued representations play an important role when 1) composite symbols are combined into more complex expressions, and 2) these embedded tensors are updated through gradient-based optimization.

尽管符号人工智能采用以语言为中心的设计，但将每个交互建模和操纵为符号表示并不是固有的高效。因此，我们为感官输入和非离散元素建立符号与子符号表示之间的映射。这种映射通常通过函数逼近实现。这使我们能够在模态到语言和语言到模态的用例之间进行映射。在这里，模态作为文本、图像、视频、音频、运动等各种类型的占位符。反过来，每个符号对象包含值和向量值表示，这些表示通过值和嵌入属性获得。后者表示符号的当前值，类似于将文本嵌入并存储为PyTorch张量（Paszke等，2019）或NumPy数组（Harris等，2020）。虽然对于大型语言模型（LLM）而言，数值张量可能缺乏固有意义，但向量值表示在1）复合符号组合成更复杂的表达式时，以及2）这些嵌入张量通过基于梯度的优化进行更新时，发挥着重要作用。

To enable the processing of symbols by LLMs, we assume that each Symbol object implements Python's native string functionality,where the "_str_ method returns an interpretable string representation. Therefore, we can assert that any Python object is parsable by an LLM, however, the user must ensure a meaningful representation. For more details, see Appendix Section E.

为了使大型语言模型（LLMs）能够处理符号，我们假设每个符号对象实现了Python的原生字符串功能，其中“_str_”方法返回可解释的字符串表示。因此，我们可以断言任何Python对象都可以被LLM解析，然而，用户必须确保有意义的表示。有关更多详细信息，请参见附录E节。

Polymorphic Context Polymorphism is a central concept in programming language theory and prominently featured in SymbolicAI. Polymorphism refers to the ability of different objects to be accessed through the same interface, or of a single identifier to represent different types based on the context of execution. Providing a single interface for entities of different types allows operations to be performed in ways specific to their derived types. We therefore designed the Symbol object to contain a global context, which is composed of static and dynamic context parts, and enables this polymorphic behavior. The static context is class dependent and defined at design time. The dynamic context is runtime adaptable and can be changed to adhere to runtime specific logic and changes. Moreover, Symbol associated operations are resolved following polymorphic design before being evaluated by the NeSy engine. Symbol-icAI's engine implementation contains a prepare method to resolve and compile the engine specific representation by evaluating the Symbol-specific operations and context. For an example on polymorphic context see part a) in Figure 3.

多态上下文多态性是编程语言理论中的一个核心概念，并在符号人工智能中占据重要地位。多态性指的是不同对象能够通过相同接口进行访问的能力，或者单一标识符能够根据执行上下文表示不同类型的能力。为不同类型的实体提供单一接口，使得可以以特定于其派生类型的方式执行操作。因此，我们设计了符号对象，以包含一个由静态和动态上下文部分组成的全局上下文，从而实现这种多态行为。静态上下文依赖于类，并在设计时定义。动态上下文在运行时可适应，并可以根据运行时特定逻辑和变化进行更改。此外，符号相关的操作在被NeSy引擎评估之前，按照多态设计进行解析。Symbol-icAI的引擎实现包含一个准备方法，通过评估符号特定的操作和上下文来解析和编译引擎特定的表示。有关多态上下文的示例，请参见图3的a)部分。

Operators and Methods In SymbolicAI, operators are overloaded to facilitate transformations of Symbol objects. These operator primitives employ dynamic casting to ensure type compatibility. Consequently, Symbol objects can be easily manipulated through type specific attributions or symbolically evaluated by the NeSy engine. For example, primarily adhere to the type specific implementation, because we prioritize strict comparisons over probabilistic evaluations. If the evaluation was unsuccessful, we then consider semantic equality through the NeSy engine. SymbolicAI leverages decorators for composing operators and custom class methods. For more details, see Appendix Section C.

运算符和方法在符号人工智能中，运算符被重载以促进符号对象的转换。这些运算符原语采用动态类型转换以确保类型兼容性。因此，符号对象可以通过特定类型的属性轻松操作，或通过NeSy引擎进行符号评估。例如，主要遵循特定类型的实现，因为我们优先考虑严格比较而非概率评估。如果评估不成功，我们会通过NeSy引擎考虑语义相等性。符号人工智能利用装饰器来组合运算符和自定义类方法。有关更多详细信息，请参见附录C节。

Upon invoking an operator or method, the respective primitive function evaluates the symbol's specific type and its respective attributes, and if necessary, resolves a nested decorated function that then uses the NeSy engine for evaluation. Should the evaluation fail, a predefined fallback implementation executes. Absent a fallback, or if both evaluations fail, an error state is raised. The processing of an operator or custom method involves a pipeline consisting of pre- and post-processing steps, as well as constraint enforcement. Constraints cover aspects like return types, value ranges, and structural integrity (e.g. JSON formatting through grammar-based verification). In Figure 3 b) we give an overview of the entire prompt composition based on the user input, the Symbol object structure, and in part c) the Symbol evaluation pipeline.

在调用操作符或方法时，相应的原始函数会评估符号的特定类型及其相应属性，并在必要时解析一个嵌套的装饰函数，该函数随后使用 NeSy 引擎进行评估。如果评估失败，则执行预定义的回退实现。如果没有回退，或者如果两个评估都失败，则会引发错误状态。操作符或自定义方法的处理涉及一个管道，该管道包括前处理和后处理步骤，以及约束执行。约束涵盖返回类型、值范围和结构完整性（例如，通过基于语法的验证进行 JSON 格式化）。在图 3 b) 中，我们提供了基于用户输入、符号对象结构的整个提示组成的概述，在部分 c) 中则展示了符号评估管道。

Self-Referential Structures SymbolicAI augments the generative process by enabling systems to introspect and modify their behavior dynamically. We leverage LLMs to execute tasks based on both natural and formal language instructions, adhering to the specified user objectives and with innate self-referential structures. We derive subtypes from Expression and enclose their functionalities in task-specific components, which we then expose again through

自指结构 SymbolicAI 通过使系统能够动态地自省和修改其行为来增强生成过程。我们利用大型语言模型（LLMs）根据自然语言和形式语言指令执行任务，遵循指定的用户目标，并具备内在的自指结构。我们从表达式派生子类型，并将其功能封装在特定任务的组件中，然后再次通过这些组件暴露出来。

Figure 3: a) Illustration of polymorphic context on the example of a SQLExpression type for the add-operator. Without a polymorphic context a regular Expression evaluation concatenates two Symbol objects together. The polymorphic context in SQLExpression overwrites the base behavior such that two added SQL-expressions get semantically combined, not concatenated. b) Illustration of the translation of a Symbol object to a prompt statement to be processed by an LLM in the NeSy engine. The User Input Args can be attached with a Payload from previous executions and gets applied to the Custom Method. The user input with the polymorphic context of the Symbol Object attributes (Static Context and Dynamic Context) are translated to a prompt statement according to the schema of the Prompt Design. The fields Operation, Examples and Template mark operation description, DSL-based prompt examples and template structures respectively. These translations are processed according to PreProcessor and engine-specific formatting. c) Illustrates the evaluation pipeline from user input to output, with multiple translation processes before and after the Engine invocation. The Input gets passed to the Custom Method and reformatted according to a PreProcessor to adhere to DSL-specific structure. The engine then takes the output of the PreProcessor and composes the final prompt according to the engine-specific Prompt Design and resolves polymorphic context and auxiliary fields. The output of the Engine then can be restructured by a Post Processor to match DSL-requirements of the desired Output and gets applied Constraints to verify the outcome.

图3：a) 以 SQLExpression 类型的加法运算符为例，展示多态上下文的插图。没有多态上下文时，常规的表达式评估将两个符号对象连接在一起。SQLExpression 中的多态上下文覆盖了基本行为，使得两个相加的 SQL 表达式在语义上被组合，而不是简单地连接在一起。b) 展示符号对象转换为提示语句以供 NeSy 引擎中的 LLM 处理的插图。用户输入参数可以附带来自先前执行的有效负载，并应用于自定义方法。用户输入与符号对象属性的多态上下文（静态上下文和动态上下文）被翻译为根据提示设计的模式生成的提示语句。操作、示例和模板字段分别标记操作描述、基于 DSL 的提示示例和模板结构。这些翻译根据预处理器和引擎特定格式进行处理。c) 说明从用户输入到输出的评估管道，在引擎调用之前和之后有多个翻译过程。输入被传递给自定义方法，并根据预处理器重新格式化，以遵循特定于 DSL 的结构。然后引擎获取预处理器的输出，并根据引擎特定的提示设计组合最终提示，同时解析多态上下文和辅助字段。引擎的输出可以通过后处理器重新结构化，以符合所需输出的 DSL 要求，并应用约束以验证结果。

templating and the model-driven design of the NeSy engine. This design choice allows a system to create and utilize its own sub-process definitions, analogous to concepts discussed in Schmidhuber (2007; 2009). Concretely, we utilize generalization properties of LLMs to interpret and formulate a set of operations that incorporate self-instructions (Wang et al., 2022). Consequently, the operations hold the flexibility to adapt to the context, and derive sub-processes that self-instruct LLMs to engage in situational modeling and context-sensitive problem-solving. Ultimately, this enables the construction of hierarchical computational graphs for self-referential meta-reasoning systems without the need to explicitly training a meta-learner (Kirsch & Schmidhuber, 2022). In Figure 4 we illustrate a step-wise evaluation of a contextual computational graph, in which the NeSy engine is processing conditioned on the current execution context and producing a next symbol prediction.

模板化和NeSy引擎的模型驱动设计。这一设计选择允许系统创建和利用其自身的子过程定义，类似于Schmidhuber（2007; 2009）中讨论的概念。具体而言，我们利用大型语言模型（LLMs）的泛化特性来解释和制定一组操作，这些操作包含自我指令（Wang et al., 2022）。因此，这些操作具有适应上下文的灵活性，并推导出自我指令LLMs参与情境建模和上下文敏感问题解决的子过程。最终，这使得构建自指元推理系统的层次计算图成为可能，而无需显式训练元学习者（Kirsch & Schmidhuber, 2022）。在图4中，我们展示了上下文计算图的逐步评估，其中NeSy引擎根据当前执行上下文进行处理，并生成下一个符号预测。

One of the challenges when creating multi-step generative processes with LLMs as part of NeSy engines relies on model evaluation and handling irrelevant predictions. The naive assessment that measures only task succession would score all models to zero and render them as unusable. Even if models follow instructions and produce parts of the expected solution, we regularly observe that they - especially open-source models - append a continuation of task irrelevant predictions. Such predictions result in failure modes when applying conditions and validations, and halt any more nuanced diagnostics and the possibility of continuing the evaluation despite intermediate failures. To derive our quality measure, we borrow ideas from the utilization of the Fréchet distance for generative processes (Heusel et al.,

在将LLMs作为NeSy引擎的一部分创建多步骤生成过程时，面临的挑战之一是模型评估和处理无关预测。仅仅测量任务连续性的简单评估将所有模型的得分都归为零，使其变得不可用。即使模型遵循指令并生成预期解决方案的部分，我们经常观察到它们——尤其是开源模型——附加了与任务无关的预测的延续。这些预测在应用条件和验证时导致失败模式，并阻碍任何更细致的诊断以及尽管存在中间失败仍继续评估的可能性。为了推导我们的质量测量，我们借鉴了在生成过程中使用Fréchet距离的思想（Heusel et al.,

Figure 4: We showcase a multi-step hierarchical computational graph, with each node in the graph represented by a symbol. The edges are relations between symbols. The left-hand side illustrates how a new node (Symbol 3) is obtained by evaluating an operation with its respective context on a NeSy engine. The right-hand side illustrates the context information window (yellow rectangle) and relationship of the resulting graph with its respective nodes.

图4：我们展示了一个多步骤的层次计算图，图中的每个节点由一个符号表示。边表示符号之间的关系。左侧展示了如何通过在NeSy引擎上评估一个操作及其相应上下文来获得一个新节点（符号3）。右侧展示了上下文信息窗口（黄色矩形）以及生成图与其相应节点的关系。

We generate trajectories through a NeSy sequential process that creates a trajectory of distributions

P

over multiple desired behavior. To quantify the validity of the generated trajectories, we measure the total distance between th generated and reference data distribution along the path trajectory. We therefore adopt a cumulative measure capable of taking into account the entire generative trajectory. In theory, this process would entail calculating the path integral over the latent space representations for models, cumulating the Fréchet distances (Dowson & Landau, 1982) traversed

我们通过NeSy顺序过程生成轨迹，该过程在多个期望行为上创建分布的轨迹

P

。为了量化生成轨迹的有效性，我们测量生成的和参考数据分布沿路径轨迹的总距离。因此，我们采用一种累积度量，能够考虑整个生成轨迹。从理论上讲，这一过程将涉及计算模型的潜在空间表示上的路径积分，累积经过的Fréchet距离（Dowson & Landau, 1982）。

where

D (P_{gen}, P_{ref})

denotes the integral of the Fréchet distances between two data distributions along the generative path trajectory from an initial time

t_{0}

to a final time

t_{f}, d (N (m_{t}, C_{t}), N (m_{w, t}, C_{w, t}))

is the Fréchet distance calculated at each time

t

between the generated multivariate normal data distribution with mean

m_{t}

and covariance

C_{t}

, and the reference multivariate normal data distribution with mean

m_{w, t}

and covariance

C_{w, t}

. The resulting measure follows properties of normal distributions and is consistent with increasing disturbances.

其中

D (P_{gen}, P_{ref})

表示沿着从初始时间

t_{0}

到最终时间

t_{f}, d (N (m_{t}, C_{t}), N (m_{w, t}, C_{w, t}))

的生成路径轨迹之间两个数据分布的Fréchet距离的积分

t

是在每个时间点计算的Fréchet距离，生成的多元正态数据分布的均值为

m_{t}

，协方差为

C_{t}

，而参考的多元正态数据分布的均值为

m_{w, t}

，协方差为

C_{w, t}

。所得度量遵循正态分布的特性，并与增加的干扰一致。

However, this approach is computationally intractable for large-scale problems, and requires access to latent representations, which - especially in the context of LLMs - is not always given. For computational feasibility, we introduce an approximation that measures the embedding distances over the path trajectories through an auxiliary embedding model, based on prior work on distribution regression (Szabó et al., 2016). The embedding model maps the symbolic representations into a RKHS, such that we can apply a kernel mean embedding function to measure their respective distances (You et al., 2019; Dinu et al., 2023). We assess the distance through the mean embeddings w.r.t. to a kernel function

K (\cdot, \cdot)

of the samples

e_{x}^{t} \sim ν_{gen}^{t} \in P_{gen}

and

e_{y}^{t} \sim ν_{ref}^{t} \in P_{ref}

produced by the generated data distribution and a reference data distribution respectively. We denote by

μ_{e_{x}^{t}}, μ_{e_{y}^{t}}

the mean embeddings associated to the respective samples,i.e.

μ_{e_{x}^{t}} (z) = \frac{1}{n} \sum_{i = 1}^{n} K (x_{i}^{t}, z)

in case

e_{x}^{t} = {(x_{i}^{t})}_{i = 1}^{n}

is a sample of size

n

of the respective mean embed-dings. To compute the similarity between the embeddings of the generated and reference distributions, we evaluate the associated maximum mean discrepancy

{MMD}^{2} (μ_{e_{x}^{t}}, μ_{e_{y}^{t}})

(Gretton et al.,2012) and then,as before for the Fréchet

然而，这种方法在大规模问题上计算上是不可行的，并且需要访问潜在表示，这在大型语言模型（LLMs）的上下文中并不总是可得。为了实现计算的可行性，我们引入了一种近似方法，通过辅助嵌入模型测量路径轨迹上的嵌入距离，这基于先前关于分布回归的工作（Szabó et al., 2016）。嵌入模型将符号表示映射到再生核希尔伯特空间（RKHS），使我们能够应用核均值嵌入函数来测量它们各自的距离（You et al., 2019; Dinu et al., 2023）。我们通过相对于样本

e_{x}^{t} \sim ν_{gen}^{t} \in P_{gen}

和

e_{y}^{t} \sim ν_{ref}^{t} \in P_{ref}

生成的数据分布及参考数据分布的核函数

K (\cdot, \cdot)

的均值嵌入来评估距离。我们用

μ_{e_{x}^{t}}, μ_{e_{y}^{t}}

表示与各自样本相关的均值嵌入，即在

e_{x}^{t} = {(x_{i}^{t})}_{i = 1}^{n}

是各自均值嵌入的大小为

n

的样本的情况下。为了计算生成分布和参考分布的嵌入之间的相似性，我们评估相关的最大均值差异

{MMD}^{2} (μ_{e_{x}^{t}}, μ_{e_{y}^{t}})

（Gretton et al., 2012），然后像之前对 Fréchet 的处理一样。

the properties of the MMD and derive a similarity score, which follows the same statistical principles as the MMD, and is bound between

[0, 1]

. We concluded that we can utilize only the MMD cross terms to evaluate the similarities. See Appendix Section B for more details. For our comparisons as referenced in Figure 6 we therefore denote the similarities rather than distances. We then come to the following formulation and refer to our empirical measure as the "Vector Embedding for Relational Trajectory Evaluation through Cross-similarity", or VERTEX score for short:

MMD 的性质并推导出一个相似性评分，该评分遵循与 MMD 相同的统计原理，并且范围在

[0, 1]

之间。我们得出结论，只有 MMD 交叉项可以用于评估相似性。有关更多详细信息，请参见附录 B 节。因此，在我们在图 6 中提到的比较中，我们表示相似性而不是距离。然后我们得出以下公式，并将我们的经验度量称为“通过交叉相似性进行关系轨迹评估的向量嵌入”，简称为 VERTEX 评分：

We approximate the integral across time steps through Monte Carlo approximation. The introduced normalization constants denote the similarities to a random sequence

z_{rand}

,which functions as a baseline subtraction to recenter our results,and a given reference score to rescale w.r.t. to scores obtained from comparing related solutions

z

. Min-max scaling ensures the final measure is bounded between

[0, 1]

. This process reflects properties such as Hölder continuity that bounds the kernel function within certain limits. To compute the embeddings, we utilize the embedding model all-mpnet-base-v2 (Song et al., 2020), due to its widespread availability, and its balance between speed and quality. As a similarity measure, we select a Gaussian kernel following our derivation from the Appendix Section B. In our implementations, we also explore other kernels, including preliminary experiments with cosine similarity. We also note that one can integrate Bernoulli distributed trials into our score, with 0 values representing failure modes and values of 1 being successes. Furthermore, if we relax our definition, we can integrate other similarity measures which are bound between

[0, 1]

,which then reflect on domain-specific attributions,i.e. including a similarity measure tailored towards capturing the nuances between two sub-structures of abstract syntax tree.

我们通过蒙特卡洛近似法来近似时间步长上的积分。引入的归一化常数表示与随机序列的相似性

z_{rand}

，该序列作为基线减法来重新中心我们的结果，并且给定的参考分数用于相对于从比较相关解决方案中获得的分数进行重新缩放

z

。最小-最大缩放确保最终度量在

[0, 1]

之间有界。这个过程反映了如 Hölder 连续性等性质，这些性质在某些限制内界定了核函数。为了计算嵌入，我们利用了嵌入模型 all-mpnet-base-v2 (Song et al., 2020)，因为它的广泛可用性以及在速度和质量之间的平衡。作为相似性度量，我们选择了高斯核，依据我们在附录 B 部分的推导。在我们的实现中，我们还探索了其他核，包括与余弦相似度的初步实验。我们还注意到，可以将伯努利分布的试验整合到我们的分数中，其中 0 值表示失败模式，1 值表示成功。此外，如果我们放宽定义，可以整合其他在

[0, 1]

之间有界的相似性度量，这些度量反映了特定领域的属性，即包括一种旨在捕捉抽象语法树两个子结构之间细微差别的相似性度量。

We introduce a benchmark that evaluates multi-step generative processes as complex workflows. Our benchmark consists of five different evaluation categories, and uses the VERTEX score to measure the capabilities of an LLM to solve tasks from each category. The five categories of our benchmark are divided into three basic evaluations and two tion which measures a models proficiency in understanding associations between symbols, (ii) multi-modal binding where we evaluate the capability to employ tools and operate on different modalities, and (iii) program synthesis for measuring a models proficiency in generating consistent code and executing subroutines. The two advanced benchmark categories are (iv) logic, for evaluating logic-based components and (v) computational graphs where complex workflows need to be processed, evaluating all aforementioned capabilities. For our evaluation we focus on the GPT family (Brown et al., 2020) of models, specifically GPT-3.5 Turbo (revision 1106) and GPT-4 Turbo (revision 1106) as they are the most proficient models to date; Gemini-Pro (Google, 2023) as the best performing model available through API from Google; LLaMA2-Chat 13B (Touvron et al., 2023), LLaMA3-Chat 8B and LLaMA3-Chat 70B 2023) serve as baselines for revised and fine-tuned open-source models. The open-source models Mistral, Zephyr, and smaller LLaMA variants are estimated to have roughly equivalent parameter counts compared to GPT-3.5 Turbo and Gemini-Pro. All our experiments require a context size smaller or equal to 4096 to enable the comparisons among the in-context capabilities across model architectures. For the LLaMA models, we utilize the chat versions since they are specifically tuned to follow instructions.

我们介绍了一个基准，评估多步骤生成过程作为复杂工作流。我们的基准由五个不同的评估类别组成，并使用 VERTEX 分数来衡量大型语言模型（LLM）解决每个类别任务的能力。我们的基准的五个类别分为三个基本评估和两个高级评估，其中 (i) 符号关联理解能力的评估，(ii) 多模态绑定能力的评估，我们评估使用工具和操作不同模态的能力，以及 (iii) 程序合成，用于衡量模型生成一致代码和执行子例程的能力。两个高级基准类别是 (iv) 逻辑，用于评估基于逻辑的组件，以及 (v) 计算图，其中需要处理复杂工作流，评估所有上述能力。对于我们的评估，我们专注于 GPT 家族（Brown et al., 2020）的模型，特别是 GPT-3.5 Turbo（修订版 1106）和 GPT-4 Turbo（修订版 1106），因为它们是迄今为止最优秀的模型；Gemini-Pro（Google, 2023）作为通过 Google API 提供的最佳表现模型；LLaMA2-Chat 13B（Touvron et al., 2023）、LLaMA3-Chat 8B 和 LLaMA3-Chat 70B（2023）作为修订和微调的开源模型的基准。开源模型 Mistral、Zephyr 和较小的 LLaMA 变体的参数数量估计与 GPT-3.5 Turbo 和 Gemini-Pro 大致相当。我们所有的实验都要求上下文大小小于或等于 4096，以便在模型架构之间进行上下文能力的比较。对于 LLaMA 模型，我们使用聊天版本，因为它们经过专门调优以遵循指令。

Associative Prediction We evaluate a model's proficiency to follow simple and complex instructions and associations with zero- and few-shot examples. Therefore, we evaluate the proficiency in applying our operators between Symbol types. We defined a total of 15 tasks involving in-context associations between two Symbol instances. SymbolicAI's overloaded operators rely on predefined pseudo-grammars, as described in Section 4, that augment the operators with few-shot examples. For instance, the overloaded operator + utilized between two Symbol instances provides few-shot examples how to resolve additions with various data types. Consequently, we can now test if the models can solve the addition between Symbol ("two hundred and thirty four") and Symbol (7000). See Appendix Section F. 1 for more details.

关联预测我们评估模型遵循简单和复杂指令及其与零样本和少量样本示例的关联能力。因此，我们评估在符号类型之间应用我们操作符的能力。我们定义了总共15个任务，涉及两个符号实例之间的上下文关联。SymbolicAI的重载操作符依赖于预定义的伪语法，如第4节所述，这些伪语法通过少量示例增强了操作符。例如，在两个符号实例之间使用的重载操作符 + 提供了如何使用各种数据类型解决加法的少量示例。因此，我们现在可以测试模型是否能够解决符号（“二百三十四”）和符号（7000）之间的加法。有关更多详细信息，请参见附录F.1节。

Multi-modal Binding We perform transformations between multiple modalities through language-based representations. Therefore, we need to evaluate the model's proficiency in tool utilization, classification and routing of requests to relevant modules. We define a multi-modal Expression to detect the category of a task based on its content and to forward the task to the appropriate tool. The expression creates interfaces to tools like WolframAlpha for mathematical expressions, Selenium for website content scraping, SerpApi for search queries, and APILayer for optical character recognition. Each of the five tests aims to evaluate the appropriate handling of a specific type of input by the multi-modal Expression type, such as processing a website URL for scraping, interpreting a search engine query, testing if two vectors are linearly independent, comparing large numbers, and extracting text from an image. See Appendix Section F. 2 for more details.

多模态绑定我们通过基于语言的表示在多种模态之间进行转换。因此，我们需要评估模型在工具利用、请求分类和路由到相关模块方面的能力。我们定义了一个多模态表达式，以根据任务的内容检测其类别，并将任务转发到适当的工具。该表达式创建了与工具的接口，如用于数学表达式的WolframAlpha、用于网站内容抓取的Selenium、用于搜索查询的SerpApi，以及用于光学字符识别的APILayer。五个测试的每一个旨在评估多模态表达式类型对特定类型输入的适当处理，例如处理用于抓取的网站URL、解释搜索引擎查询、测试两个向量是否线性独立、比较大数字，以及从图像中提取文本。有关更多详细信息，请参见附录F.2节。

Program Synthesis We evaluate executable code with and without concepts from retrieval augmented generation, model-driven development, and experiment with self-generated instructions by creating self-referential expressions. We designed three separate tests related to program synthesis, where each task assesses the ability of the models to generate and execute code based on natural language instructions or provided templates:

程序合成我们评估可执行代码，包括和不包括检索增强生成、模型驱动开发的概念，并通过创建自指表达式来实验自生成的指令。我们设计了三个与程序合成相关的独立测试，每个任务评估模型根据自然语言指令或提供的模板生成和执行代码的能力：

Logical Components To evaluation the capabilities for logical reasoning of models, we condition them to create a sequence of expressions as self-contained components, and refer to higher-order logic for their assessment. Based on the underlying type theory originating from Whitehead & Russell (1925-1927), we evaluate a models' capability to resolve statements in the form of there exists

x

such that

x

fulfills

y

. Such quantifiers define the standard semantics of expressions, where their meaning is given by a semantic function. A semantic function maps a term from an abstract definition to a point in a domain, which is an interpretation of the term's type and value. Therefore, these functions operate on types and values of expressions, and relations thereof. Subsequently, NeSy engines can formulate and evaluate at inference time logic-based instructions through Lisp, Prolog, or Mathematica (McCarthy, 1959; Colmerauer & Roussel,1993; Chen et al.,1993; Inc.,2022), or leverage solvers such as Z3 (Moura & Bjørner, 2008). Therefore, the result of a natural language statement when evaluated by a NeSy engine can be interpreted by any expert system which defines the corresponding semantic functions and process them either in a symbolic (Feigenbaum et al., 1965; Gamble et al., 1994), differentiable (Veličković & Blundell, 2021; Ibarz et al., 2022), or hybrid manner (Kuncicky et al., 1991).

逻辑组件为了评估模型的逻辑推理能力，我们将其条件化以创建一系列作为自包含组件的表达式，并参考高阶逻辑进行评估。基于源自怀特海和拉塞尔（1925-1927）的基础类型理论，我们评估模型解决以“存在某个

x

使得

x

满足

y

”形式的陈述的能力。这种量词定义了表达式的标准语义，其意义由语义函数给出。语义函数将一个术语从抽象定义映射到域中的一个点，这个点是该术语类型和价值的解释。因此，这些函数在表达式的类型和值及其关系上进行操作。随后，NeSy 引擎可以通过 Lisp、Prolog 或 Mathematica（麦卡锡，1959；科尔梅劳尔和鲁塞尔，1993；陈等，1993；公司，2022）在推理时制定和评估基于逻辑的指令，或利用如 Z3（穆拉和比约尔纳，2008）这样的求解器。因此，当 NeSy 引擎评估自然语言陈述的结果时，可以被任何定义相应语义函数并以符号（费根鲍姆等，1965；甘布尔等，1994）、可微分（维利克诺维奇和布伦德尔，2021；伊巴兹等，2022）或混合方式（昆西基等，1991）处理的专家系统解释。

We evaluate how proficient models are at interpreting custom DSLs and define expression statements. DSLs are designed to express logical relations and operations in a structured format, and supports human-readable and machine-statement into an expression statement, as follows:

我们评估模型在解释自定义 DSL（领域特定语言）和定义表达语句方面的熟练程度。DSL 旨在以结构化格式表达逻辑关系和操作，并支持人类可读和机器可陈述的表达语句，如下所示：

An automated theorem prover can now evaluate this statement for all values of

x

and assess the validity of the original query. Lastly, our evaluation uses symbolic mathematics to manipulate algebraic expressions. This involves defining symbols and performing operations like factorization, simplification, and algebraic manipulation. The symbols are placeholders for any value, enabling the definition of general expressions without specifying their values upfront.

自动定理证明器现在可以评估该语句在所有

x

的值下的有效性，并评估原始查询的有效性。最后，我们的评估使用符号数学来操纵代数表达式。这涉及到定义符号并执行因式分解、简化和代数操作等操作。这些符号是任何值的占位符，使得可以定义一般表达式而无需提前指定它们的值。

2) Three tasks evaluate a models' capability to resolve the logical operations AND, OR, and XOR. For instance, the test for logical AND combines the symbols Symbol ("The horn only sounds on Sundays") and Symbol ("I hear the horn") and compares the answer against the human-generated references "The horn of possible solutions, there is high variability in the solution space. Each model might prefer a different solution.

2) 三个任务评估模型解决逻辑运算 AND、OR 和 XOR 的能力。例如，逻辑 AND 的测试结合了符号 Symbol（“喇叭只在星期天响”）和符号 Symbol（“我听到喇叭声”），并将答案与人类生成的参考“可能解决方案的喇叭”进行比较，解决方案空间中存在高度变异性。每个模型可能偏好不同的解决方案。

3) For another task we use a custom Expression that defines a DSL syntax and semantic structure. We use this Expression to extract higher-order logic expressions from a natural language statement, namely the puzzle 'Who is Jay’s brother?’

^{4}

,that preserves the original relationships.

3) 在另一个任务中，我们使用自定义表达式定义 DSL 语法和语义结构。我们使用该表达式从自然语言语句中提取高阶逻辑表达式，即谜题“杰伊的兄弟是谁？”

^{4}

，以保持原始关系。

4) For the final task, we again use the puzzle 'Who is Jay's brother?' to evaluate a models' capability for complex conversions. We use the Z3 theorem prover (Moura & Bjørner, 2008) to solve the 'Who is Jay's brother' puzzle conditioned on the Z3 solvers' solution to Einsteins' famous puzzle 'Who owns the fish?'. The task involves an indirect translation from natural language to executable code by the Z3 solver; the solution to Einstein's puzzle acts as a form of self-contained "documentation" for how the Z3 solver should be utilized. The test constructs a template, which includes the task instructions, puzzle statement, and reference to the Einstein's puzzle solution. The models are then asked to analyze the given problem and solution format and create a Python function with Z3 syntax that can solve the 'Who is Jay's brother?' puzzle. The dynamically generated code is executed within the test environment utilizing Python's exec function. We check the access to the Z3 solver and run the generated solve_puzzle function supposed to contain the logic to solve the puzzle. Once executed, the assembled Z3 logical clauses are processed by the solver, which verifies that the set of constraints is satisfiable. If so, the model generated by the solver is queried for the puzzle's solution and scored using our VERTEX score.

4) 对于最终任务，我们再次使用谜题“谁是杰伊的兄弟？”来评估模型在复杂转换方面的能力。我们使用 Z3 定理证明器 (Moura & Bjørner, 2008) 来解决“谁是杰伊的兄弟”谜题，该谜题以 Z3 求解器对爱因斯坦著名谜题“谁拥有鱼？”的解答为条件。该任务涉及从自然语言到可执行代码的间接翻译，由 Z3 求解器完成；爱因斯坦谜题的解答作为一种自包含的“文档”，说明 Z3 求解器应如何使用。测试构建了一个模板，其中包括任务说明、谜题陈述和对爱因斯坦谜题解答的引用。然后，要求模型分析给定的问题和解答格式，并创建一个具有 Z3 语法的 Python 函数，以解决“谁是杰伊的兄弟？”的谜题。动态生成的代码在测试环境中利用 Python 的 exec 函数执行。我们检查对 Z3 求解器的访问，并运行生成的 solve_puzzle 函数，该函数应包含解决谜题的逻辑。一旦执行，组装的 Z3 逻辑子句将由求解器处理，求解器验证约束集是否可满足。如果可满足，则查询求解器生成的模型以获取谜题的解答，并使用我们的 VERTEX 评分进行评分。

Hierarchical Computational Graphs We evaluate the capabilities of models to orchestrate a multi-step generative process and evaluate a set of tasks. Models need to direct sub-processes and associate computational results from and to Symbol nodes, and maintain relationships between these nodes, which we refer to as a computational graph as shown in Figure 5. In a computational graph, the VERTEX score compares the results produced by a generative model at each node against samples obtained from a reference distribution, usually modeled by sampling from multiple valid references. We also account for randomness through predefined random samples for normalizing the result. Our reference to hierarchical computational graphs stems from the fact that we operate on multiple levels. On a higher level of abstraction we are able to perform planning, sub-task scheduling, and define operational instructions. On a lower level of abstraction, we execute these plans based on the defined instructions and data, which can also span generative processes that produce new information.

层次计算图我们评估模型在协调多步骤生成过程和评估一组任务方面的能力。模型需要指导子过程，并将计算结果与符号节点进行关联，同时维护这些节点之间的关系，我们称之为计算图，如图5所示。在计算图中，VERTEX评分比较生成模型在每个节点产生的结果与从参考分布中获得的样本，通常通过从多个有效参考中采样来建模。我们还通过预定义的随机样本来考虑随机性，以规范化结果。我们对层次计算图的引用源于我们在多个层次上操作的事实。在更高的抽象层次上，我们能够进行规划、子任务调度，并定义操作指令。在较低的抽象层次上，我们根据定义的指令和数据执行这些计划，这些数据也可以跨越生成过程，产生新信息。

Figure 5: We illustrate the hierarchical computational graph for the Paper expression. Each node represents an instance of an expression with distinct properties and behaviors, such as file sourcing, generative process, tool utilization, or transformation operation. The edges denote the reference relationships between expressions and indicate the flow of information. The blue highlighted nodes mark the main sequence nodes of expressions utilized to create parts of the paper draft, such as Method section, RelatedWork section, Abstract section, and so on. Each generative node is used for evaluating the VERTEX score. None-generative nodes such as search engine results are not evaluated, and we assume to obtain ground-truth values.

图5：我们展示了论文表达的层次计算图。每个节点代表具有不同属性和行为的表达实例，例如文件来源、生成过程、工具利用或转换操作。边缘表示表达之间的参考关系，并指示信息流动。蓝色高亮的节点标记了用于创建论文草稿部分的主要序列节点，例如方法部分、相关工作部分、摘要部分等。每个生成节点用于评估VERTEX评分。非生成节点，如搜索引擎结果，则不进行评估，我们假设获得真实值。

1) We simulate and evaluate the process of writing a research paper draft based on a predefined hierarchical computational graph that focuses on the content output of the computational graph rather than planning and scheduling functionality. See Appendix Section F. 5 for more details.

1) 我们基于预定义的层次计算图模拟和评估撰写研究论文草稿的过程，该计算图侧重于计算图的内容输出，而不是规划和调度功能。有关更多细节，请参见附录 F. 5 节。

2) We test the VERTEX Protocol as defined in Algorithm 1, which represents our general method for evaluating multi-step generative processes. We create a self-contained test scenario to illustrate an end-to-end evaluation and as a go-to reference for how our protocol can be deployed in a realistic environment. Our evaluation protocol is not only designed to analyze and score a series of instructions, but also to provide a structured basis for recording these

2) 我们测试算法 1 中定义的 VERTEX 协议，该协议代表我们评估多步骤生成过程的一般方法。我们创建一个自包含的测试场景，以说明端到端评估，并作为我们协议在现实环境中部署的参考。我们的评估协议不仅旨在分析和评分一系列指令，还提供记录这些过程的结构化基础。此外，我们注意到我们的评估协议是一般性制定的，这允许应用非顺序的规划和调度。算法 1 VERTEX 协议

Require: NeSy engine

V

as an LLM,embedding engine

E : \sum \to H \subset R^{d}

,symbols

{x_{0}, x^{*}, y^{*}} \subset \sum

,with

x_{0}

as the initial instruction,

x^{*}

as the payload resulted from executing

V, y^{*}

as the reference,and

*

acting as a placeholder for

P, T, C

,capabilities

C = {F_{1}, F_{2}, F_{3}, \dots}

,where each

F_{i}

represents a specific functional role within the system,plan

P \subset \sum

,task

T \in P

,memory buffer

M \subset \sum

,a scoring function

\tilde{s} : H \times H \to [0, 1]

,a scheduler

Q

, an aggregator

A

,and score variables

{s} \in [0, 1]

. Method: 1:

V, E, Q, C, y^{P} \leftarrow INIT (\cdot)

⊳

Initialize engines,scheduler,capabilities,expected plan. 2:

M \leftarrow \emptyset, A \leftarrow \emptyset

⊳

Initialize memory buffer and aggregator. 3:

x^{P} \leftarrow GENERATEPLAN (x_{0}, V)

⊳ V

generates plan based on initial instruction.

EVALUATE (x^{P}, y^{P}, E, A, \tilde{s})

⊳

Embed,score,and aggregate plan similarity.

P, M \leftarrow UNFOLDPLAN (y^{P}, M, Q)

⊳ Q

unfolds plan into actionable tasks and updates progression. while

P \neq \emptyset

⊳

Run until list of tasks is exhausted.

T, y^{C}, y^{T} \leftarrow SELECT (M, V)

⊳ V

selects next task based on task progression.

F_{i} \leftarrow IDENTIFY (T, C, V)

⊳ V

identifies task-related capability

F_{i}

x^{C}, x^{T} \leftarrow EXECUTE (T, F_{i}, Q)

⊳ Q

executes

T

with capability

F_{i}

and assign results

x^{C}, x^{T}

. EVALUATE

(x^{C}, y^{C}, x^{T}, y^{T}, E, A, \tilde{s}) ⊳

Embed,score,and aggregate capability similarity.

P, M \leftarrow UPDATE (T, P, M, Q)

⊳ Q

updates plan and progression. end while

s \leftarrow FINALIZE (A)

⊳

Finalize aggregation of scores. return

s

⊳

Return aggregated score of plan execution.

需求：NeSy 引擎

V

作为一个大型语言模型，嵌入引擎

E : \sum \to H \subset R^{d}

，符号

{x_{0}, x^{*}, y^{*}} \subset \sum

，以

x_{0}

作为初始指令，

x^{*}

作为执行

V, y^{*}

结果的有效载荷，以及

*

作为

P, T, C

的占位符，能力

C = {F_{1}, F_{2}, F_{3}, \dots}

，其中每个

F_{i}

代表系统内的特定功能角色，计划

P \subset \sum

，任务

T \in P

，内存缓冲区

M \subset \sum

，评分函数

\tilde{s} : H \times H \to [0, 1]

，调度器

Q

，聚合器

A

，以及评分变量

{s} \in [0, 1]

。方法：1：

V, E, Q, C, y^{P} \leftarrow INIT (\cdot)

⊳

初始化引擎、调度器、能力、预期计划。 2：

M \leftarrow \emptyset, A \leftarrow \emptyset

⊳

初始化内存缓冲区和聚合器。 3：

x^{P} \leftarrow GENERATEPLAN (x_{0}, V)

⊳ V

根据初始指令生成计划。

EVALUATE (x^{P}, y^{P}, E, A, \tilde{s})

⊳

嵌入、评分并聚合计划相似性。

P, M \leftarrow UNFOLDPLAN (y^{P}, M, Q)

⊳ Q

将计划展开为可执行任务并更新进展。当

P \neq \emptyset

执行

⊳

时，直到任务列表耗尽。

T, y^{C}, y^{T} \leftarrow SELECT (M, V)

⊳ V

根据任务进展选择下一个任务。

F_{i} \leftarrow IDENTIFY (T, C, V)

⊳ V

确定与任务相关的能力

F_{i}

。

x^{C}, x^{T} \leftarrow EXECUTE (T, F_{i}, Q)

⊳ Q

使用能力

F_{i}

执行

T

并分配结果

x^{C}, x^{T}

。评估

(x^{C}, y^{C}, x^{T}, y^{T}, E, A, \tilde{s}) ⊳

嵌入、评分并聚合能力相似性。

P, M \leftarrow UPDATE (T, P, M, Q)

⊳ Q

更新计划和进展。结束当

s \leftarrow FINALIZE (A)

⊳

最终汇总评分。返回

s

⊳

返回计划执行的聚合评分。

Algorithm 1: This algorithm defines the pseudocode of our VERTEX protocol with our respective VERTEX score as a scoring criteria. We start by initializing the NeSy engine

V

,the embedding engine

E

,the scheduler

Q

,and a set of capabilities

C

. The initial instruction

x_{0}

is utilized to generate a plan

x^{P}

through

V

. The plan and its expected outcome

y^{P}

are embedded,and their similarity is scored according to our VERTEX score and aggregated. The plan is then unfolded into actionable tasks. Each task

T

is selected and executed with the appropriate capability

C

,resulting in the capability and task results

x^{C}, x^{T}

,and expected outcomes

y^{C}, y^{T}

updated in the memory buffer

M

. The process continues, with each task's result being embedded, scored, and aggregated until the plan is complete. The final aggregated score

s

is returned,reflecting the overall effectiveness of the plan execution.

算法 1：该算法定义了我们 VERTEX 协议的伪代码，并以我们的 VERTEX 分数作为评分标准。我们首先初始化 NeSy 引擎

V

、嵌入引擎

E

、调度器

Q

和一组能力

C

。初始指令

x_{0}

被用于通过

V

生成计划

x^{P}

。该计划及其预期结果

y^{P}

被嵌入，并根据我们的 VERTEX 分数进行相似性评分和聚合。然后将计划展开为可执行任务。每个任务

T

被选择并以适当的能力

C

执行，导致能力和任务结果

x^{C}, x^{T}

以及预期结果

y^{C}, y^{T}

在内存缓冲区

M

中更新。该过程持续进行，每个任务的结果被嵌入、评分和聚合，直到计划完成。最终聚合分数

s

被返回，反映计划执行的整体有效性。

We start with a high-level workflow description which consists of a list of tasks and optionally their respective subtasks; we refer to this as the plan

P

. To perform the experiment,we utilize an expected plan

y^{P}

which was handcrafted for this evaluation. The expected plan is a queue of predefined tasks (in a particular order) that the system should follow to achieve the goal. The goal statement defines the end objective that the workflow aims to accomplish. We also have a set of plans similar to the expected plan,which are trajectories in the solution space,as well as the plan

x^{P}

that the LLM generates utilizing the GENERATEPLAN call for a specific seed. We score the predicted plan against the expected plan and the trajectories, then we continue to the next phase in which we utilize the expected plan to execute the tasks. At each step, the LLM will receive in its context the goal, the tasks, the current progress, and a query asking for the next task to execute; we refer to this as the memory buffer

M

. If the LLM is not able to predict the next task, it will return a failure, and the expected plan will be utilized to execute the next task. The LLM has access to a predefined set of capabilities

C

,specifically WolframAlpha,SerpApi,Selenium,and the LLM itself,which also represents our self-referential structure. We keep executing tasks until the queue is exhausted, and at each step, we utilize the EVALUATE call to measure the performance of the LLM with our VERTEX score. The scheduler class

Q

oversees the execution of the test workflow. It takes the setup configuration and orchestrates the linear execution of tasks, utilizing the expected plan as a reference. It maintains a pool of tasks to be executed and updates progress as tasks are completed. The UNFOLDPLAN call is a method of the scheduler class

Q

. The method calls itself recursively until there are no tasks left. The SELECT call is responsible for determining which task to execute next from a pool of remaining tasks. It utilizes the LLM through self-reflection (Shinn et al., 2023) to choose the most suitable next task based on a template that gets progressively updated in the memory buffer

M

by the UPDATE call. The IDENTIFY call uses self-reflection and similarity scoring to determine the best interface based on the task at hand, then passes aggregated assessment of the model's ability to manage and execute the workflow.

我们首先提供一个高层次的工作流程描述，其中包含任务列表及其各自的子任务（可选）；我们称之为计划

P

。为了进行实验，我们利用一个为本次评估精心设计的预期计划

y^{P}

。预期计划是一个预定义任务的队列（按照特定顺序），系统应遵循这些任务以实现目标。目标声明定义了工作流程旨在完成的最终目标。我们还有一组与预期计划类似的计划，它们是解空间中的轨迹，以及 LLM 为特定种子生成的计划

x^{P}

。我们将预测的计划与预期计划和轨迹进行评分，然后继续到下一阶段，在该阶段我们利用预期计划来执行任务。在每一步中，LLM 将在其上下文中接收目标、任务、当前进度以及询问下一个要执行的任务的查询；我们称之为记忆缓冲区

M

。如果 LLM 无法预测下一个任务，它将返回失败，预期计划将被用来执行下一个任务。LLM 可以访问一组预定义的能力

C

，具体包括 WolframAlpha、SerpApi、Selenium 以及 LLM 本身，这也代表了我们的自指结构。我们持续执行任务，直到队列耗尽，并且在每一步中，我们利用 EVALUATE 调用来测量 LLM 的表现，使用我们的 VERTEX 评分。调度器类

Q

负责监督测试工作流程的执行。它接受设置配置并协调任务的线性执行，利用预期计划作为参考。它维护一个待执行任务的池，并在任务完成时更新进度。UNFOLDPLAN 调用是调度器类

Q

的一种方法。该方法递归调用自身，直到没有剩余任务。SELECT 调用负责从剩余任务池中确定下一个要执行的任务。它通过自我反思 (Shinn et al., 2023) 来选择最合适的下一个任务，基于一个在记忆缓冲区

M

中通过 UPDATE 调用逐步更新的模板。IDENTIFY 调用利用自我反思和相似性评分来确定基于当前任务的最佳接口，然后传递模型管理和执行工作流程能力的综合评估。

Figure 6: We evaluate GPT-4 Turbo, GPT-3.5 Turbo, Gemini-1.0 Pro, LLaMA2-Chat 13B, LLaMA3-Chat 8B, LLaMA3-Chat 70B, Mistral 7B and Zephyr 7B on five benchmark categories: 1) Associative Prediction (Association) 2) Multi-modal Binding (Modality) 3) Program Synthesis (Code) 4) Functional Logic Components (Logic) and 5) Hierarchical Computational Graphs (Graphs). We denote the VERTEX scores for each category as a normalized value between 0 and 1 , where higher values are better. The VERTEX score is measured according to a reference baseline and normalized by random sequences to exclude noise and similarities among references distributions to rescale solutions. The shown scores are an average over all tests per category and across 8 different seeds per test.

图6：我们在五个基准类别上评估了GPT-4 Turbo、GPT-3.5 Turbo、Gemini-1.0 Pro、LLaMA2-Chat 13B、LLaMA3-Chat 8B、LLaMA3-Chat 70B、Mistral 7B和Zephyr 7B：1）关联预测（Association）2）多模态绑定（Modality）3）程序合成（Code）4）功能逻辑组件（Logic）和5）层次计算图（Graphs）。我们将每个类别的VERTEX得分表示为0到1之间的标准化值，其中较高的值更好。VERTEX得分是根据参考基线测量的，并通过随机序列进行标准化，以排除噪声和参考分布之间的相似性，从而重新缩放解决方案。所示得分是每个类别在所有测试中的平均值，并且每个测试跨8个不同的种子。

Benchmarks	GPT-4 Turbo	GPT-3.5 Turbo	Gemini 1.0 Pro	LLaMA 213B	LLaMA 3 8B	LLaMA 370B	Mistral 7B	Zephyr 7B	Random
Associations	0.94	0.51	0.93	0.40	0.46	0.57	0.46	0.27	0.08
Modality	0.83	0.65	0.81	0.50	0.43	0.63	0.39	0.43	0.07
Code	0.75	0.63	0.61	0.13	0.25	0.30	0.19	0.13	0.00
Logic	0.48	0.46	0.28	0.46	0.21	0.11	0.53	0.47	0.00
Graphs	0.39	0.34	0.26	0.06	0.05	0.15	0.03	0.03	0.00
Total	0.68	0.52	0.58	0.31	0.28	0.35	0.32	0.27	0.03

基准	GPT-4 Turbo	GPT-3.5 Turbo	Gemini 1.0 Pro	LLaMA 213B	LLaMA 3 8B	LLaMA 370B	Mistral 7B	Zephyr 7B	随机
关联	0.94	0.51	0.93	0.40	0.46	0.57	0.46	0.27	0.08
模态	0.83	0.65	0.81	0.50	0.43	0.63	0.39	0.43	0.07
代码	0.75	0.63	0.61	0.13	0.25	0.30	0.19	0.13	0.00
逻辑	0.48	0.46	0.28	0.46	0.21	0.11	0.53	0.47	0.00
图	0.39	0.34	0.26	0.06	0.05	0.15	0.03	0.03	0.00
总计	0.68	0.52	0.58	0.31	0.28	0.35	0.32	0.27	0.03

In this section, we address the evaluation results, auxiliary findings and limitations of SymbolicAI and the future directions we are focusing on. Some of the limitations stem from the inherent constraints of current technologies and dependencies on third-party systems. Additionally, the nuanced complexities of working with generative models presents further challenges.

在本节中，我们讨论评估结果、辅助发现和SymbolicAI的局限性，以及我们关注的未来方向。一些局限性源于当前技术的固有限制和对第三方系统的依赖。此外，使用生成模型的细微复杂性带来了进一步的挑战。

In Figure 6 we show the VERTEX score for all five evaluation categories on 8 different state-of-the-art models. We show the aggregated results per category, meaning the average score among all tests averaged per category and average across 8 different seeds per test. The VERTEX score is normalized between 0 and 1 , where higher values are better. Our score is non-linear due to its nature of using non-linear kernels, and captures semantic, ordinal and relative structures among the data samples. However, since our score is highly dependent on the quality of the underlying embedding model, it may omit to capture fine-grained syntactic differences such as 'Hello' vs 'hello'.

在图6中，我们展示了8种不同的最先进模型在五个评估类别上的VERTEX得分。我们展示了每个类别的汇总结果，这意味着在每个类别中所有测试的平均得分，以及每个测试在8个不同种子下的平均得分。VERTEX得分在0到1之间归一化，其中较高的值表示更好。由于使用非线性核的特性，我们的得分是非线性的，能够捕捉数据样本之间的语义、序数和相对结构。然而，由于我们的得分高度依赖于基础嵌入模型的质量，它可能会忽略捕捉细微的句法差异，例如“Hello”和“hello”之间的区别。

In our experiments, we have noticed that for associative predictions and multi-modal bindings, GPT-4 Turbo is on par with Gemini-1.0 Pro. Furthermore, there is still a large gap between open-source contestants such as LLaMA 3 even with 70B parameters compared to the closed-source alternatives from OpenAI and Google. For the rest of the experiments, we see that GPT-4 almost always outperforms all other models, except for the functional logic components category. Here, we analyzed results and found that the larger models sometimes take shortcuts by automatically returning the solution and answering that the task instructions are too complex for such a straight-forward puzzle query. However, we would rather state in general that for logic-based, planning and scheduling tasks all models act unreliably, even if slight performance differences between the models are seen in the plot. We believe this is in part due to lack of training data specifically for workflows, planning and scheduling tasks, and to imprecision in generating reliably structured output formats, such as custom DSLs or other custom in-context instructed formats. This also stems from their instruction fine-tuning, since most models are chat-based models and offer verbose responses which need to be suppressed or post-processed.

在我们的实验中，我们注意到在关联预测和多模态绑定方面，GPT-4 Turbo与Gemini-1.0 Pro不相上下。此外，即使是具有70B参数的开源竞争者如LLaMA 3，与OpenAI和Google的闭源替代品相比，仍然存在很大差距。在其余的实验中，我们发现GPT-4几乎总是优于其他所有模型，除了功能逻辑组件类别。在这里，我们分析了结果，发现较大的模型有时通过自动返回解决方案来走捷径，并回答任务指令对于如此简单的谜题查询来说过于复杂。然而，我们更愿意一般性地指出，对于基于逻辑的规划和调度任务，所有模型的表现都不可靠，即使在图中可以看到模型之间的轻微性能差异。我们相信，这在一定程度上是由于缺乏专门针对工作流、规划和调度任务的训练数据，以及在生成可靠结构化输出格式（如自定义DSL或其他自定义上下文指令格式）方面的不精确。这也源于它们的指令微调，因为大多数模型是基于对话的模型，提供冗长的响应，这需要被抑制或后处理。

We see similar performance between GPT-3.5 Turbo and LLaMA 3 70B except for the logical and graphs evaluations. We found that LLaMA 3 70B has a tendency to ask questions back if it does not understand the request instead of following the specified instructions provided. We assume this also stems from the chat-based instruction fine-tuning. Zephyr 7B and Mistral 7B have shown on par capabilities in functional logic components with larger models, however

我们观察到 GPT-3.5 Turbo 和 LLaMA 3 70B 之间的性能相似，除了逻辑和图形评估之外。我们发现 LLaMA 3 70B 如果不理解请求，倾向于反问，而不是遵循提供的具体指令。我们认为这也源于基于聊天的指令微调。Zephyr 7B 和 Mistral 7B 在功能逻辑组件方面表现出与更大模型相当的能力，但在程序合成和层次计算图实验中表现不佳。

Framework Since the framework interfaces with many tools and API services, it requires a substantial engineering although we support a variety of tools and frameworks like Selenium, WolframAlpha, or Z3, we only scratch the surface of these tools. Moreover, the utilization of grammar-based constraints validations is still experimental and limited in functionality for specific formats such as JSON and HTML. Finally, we encounter also challenges related to engineering parallelization and multiprocessing of prompts, since the concurrent execution is non-trivial, especially with intricacies of Python process management.

框架由于该框架与许多工具和 API 服务接口，因此需要大量的工程工作。尽管我们支持多种工具和框架，如 Selenium、WolframAlpha 或 Z3，但我们仅仅触及这些工具的表面。此外，基于语法的约束验证的使用仍然是实验性的，并且在特定格式（如 JSON 和 HTML）方面功能有限。最后，我们还面临与提示的工程并行化和多进程相关的挑战，因为并发执行并非易事，尤其是在 Python 进程管理的复杂性下。

Embedding Measure Our empirical measure is limited by the expressiveness of the embedding model and how well it captures the nuances in similarities between two representations. Furthermore, the obtained similarity scores are highly non-linear and difficult to interpret. For instance, two representations may address the same topic, such as the problem description and its respective solution, however, when measuring their similarity we obtain similarity scores of

\sim 70 %

. We normalize this by subtracting an inherent baseline and randomness effect,however,to ensure a more holistic and robust measurement we would need a significantly larger amount of baselines and experiments. Since we were very limited in the availability of development resources, and some presented models are only addressable through costly API walls. We are actively seeking sponsors to scale our solution and offer a more compelling

嵌入度量我们的经验度量受限于嵌入模型的表达能力以及它捕捉两个表示之间相似性的细微差别的能力。此外，获得的相似性分数高度非线性且难以解释。例如，两个表示可能涉及相同的主题，例如问题描述及其相应的解决方案，然而，在测量它们的相似性时，我们获得的相似性分数为

\sim 70 %

。我们通过减去固有的基线和随机性效应来进行归一化，然而，为了确保更全面和稳健的测量，我们需要显著更多的基线和实验。由于我们在开发资源的可用性上非常有限，并且一些呈现的模型仅通过昂贵的 API 墙可访问。我们正在积极寻求赞助商来扩展我们的解决方案并提供更具吸引力的。

Model Capabilities An obvious limitation revolves around the fixed context window size of the underlying language models. Despite the expansion of the context window in newer models such as GPT-4, the finite context still restricts the amount of data that can be processed in a single pass. All information outside the context needs to be added through information retrieval approaches, which come with their own challenges and limitations (Gao et al., 2023). This leads to side effects, including hallucination, given the model does not contain the necessary information to answer the prompted instruction, which makes it difficult to maintain long-term statefulness for complex reasoning tasks and computational graphs.

模型能力一个明显的限制是底层语言模型的固定上下文窗口大小。尽管在较新的模型（如 GPT-4）中扩展了上下文窗口，但有限的上下文仍然限制了在单次传递中可以处理的数据量。所有上下文之外的信息需要通过信息检索方法添加，这些方法自身也带来了挑战和限制（Gao et al., 2023）。这导致了副作用，包括幻觉，因为模型不包含回答提示指令所需的信息，这使得在复杂推理任务和计算图中保持长期状态变得困难。

Error Handling The complexity of error handling when evaluating complex expressions through function composi-tionality, especially between multiple modalities and different solvers, is another notable challenge. While SymbolicAI introduces mechanisms for error analysis and automated correction, these approaches are not infallible. They are often limited by the quality and expressiveness of the models, and the model's capacity to understand deeply nested logical constructs. We also note that for our evaluations, we disabled any remedy protocol, such as truncating prompts or retry schema.

错误处理通过函数组合性评估复杂表达式时，错误处理的复杂性，尤其是在多种模态和不同求解器之间，是另一个显著的挑战。虽然 SymbolicAI 引入了错误分析和自动修正的机制，但这些方法并非万无一失。它们通常受到模型质量和表达能力的限制，以及模型理解深度嵌套逻辑结构的能力。我们还注意到，在我们的评估中，我们禁用了任何补救协议，例如截断提示或重试方案。

Generalization This research is also limited by current LLM's capacity for reasoning and generalization. Although progress has been made, models are still prone to hallucinations and reasoning errors, especially when dealing with abstract, novel, or highly complex problem statements (Marcus, 2020). Furthermore, our framework's reliance on the model's ability to grasp the semantics of operations can be influenced by the training data and the model's innate biases and conceptual understanding (Mahowald et al., 2023). We also point out that the initial development of SymbolicAI started with the GPT family of models, and we may encounter innate biases in prompt design and expressiveness when utilizing other reference models. However, we also point out that prompt engineering instruction-based statements is not a reliable direction for improvement. We instead advocate for enhancing the resilience of models through fault is how to assess the disentanglement of evaluations of models on downstream tasks, to avoid testing on training samples, especially for closed-source solutions like GPT.

泛化本研究也受到当前大型语言模型（LLM）推理和泛化能力的限制。尽管已经取得了一定进展，但模型仍然容易出现幻觉和推理错误，特别是在处理抽象、新颖或高度复杂的问题陈述时（Marcus, 2020）。此外，我们框架对模型理解操作语义能力的依赖可能受到训练数据以及模型固有偏见和概念理解的影响（Mahowald et al., 2023）。我们还指出，符号人工智能（SymbolicAI）的初始发展始于GPT系列模型，在使用其他参考模型时，我们可能会遇到提示设计和表达能力方面的固有偏见。然而，我们也指出，基于指令的提示工程声明并不是一个可靠的改进方向。我们更提倡通过故障的弹性来增强模型的韧性，评估模型在下游任务上的解耦，以避免在训练样本上进行测试，特别是对于像GPT这样的闭源解决方案。

Interpretability and Transparency Finally, the issue of explainability and transparency in AI systems remains challenging. While SymbolicAI makes steps towards making computational processes more explicit and explainable problem. This can hinder trust and adoption in sensitive applications where interpretability of predictions is important.

可解释性和透明性最后，人工智能系统中的可解释性和透明性问题仍然具有挑战性。尽管符号人工智能在使计算过程更加明确和可解释方面迈出了步伐，但这可能会妨碍在对预测的可解释性要求较高的敏感应用中的信任和采用。

The goal for Algorithm 1 is to be utilized by an advanced learning agent. This agent, employing reinforcement learning methodologies (Ouyang et al., 2022; Li et al., 2023; Rafailov et al., 2023), could leverage our evaluation measure in existing benchmarks (Milani et al., 2020; Swazinna et al., 2022; Schweighofer et al., 2022) as a means to obtain reward signals to addresses a central problem in reinforcement learning, namely credit assignment (Sutton, 1984; Arjona-Medina et al., 2019; Holzleitner et al., 2020; Patil et al., 2020; Widrich et al., 2021; Dinu et al., 2022). Over time, it aims to develop the ability to autonomously generate its own plans, efficiently schedule tasks and subtasks, and carefully select the most suitable tools for each task. Our protocol lays the groundwork for this agent to learn and expand its base set of capabilities (Amaro et al., 2023), moving towards more sophisticated, self-referential orchestration of multi-step tasks. We've already noticed that research is shifting towards this type of methodology (Yuan et al., 2024). Furthermore, in Section 7 we've only considered a sequential scheduler. However, our objective is to ultimately assess a non-sequential task execution model, allowing for dynamic insertion and out-of-sequence task execution. In addition, we are interested into exploring similarities of our work with Generative Flow Networks (Bengio et al., 2021a;b; Lahlou et al., 2023). Lastly, we also discuss limitations in Appendix Section 8.2 with further opportunities for future improvements.

算法 1 的目标是被一个先进的学习代理所利用。该代理采用强化学习方法（Ouyang et al., 2022; Li et al., 2023; Rafailov et al., 2023），可以利用我们在现有基准中的评估指标（Milani et al., 2020; Swazinna et al., 2022; Schweighofer et al., 2022）作为获取奖励信号的手段，以解决强化学习中的一个核心问题，即信用分配（Sutton, 1984; Arjona-Medina et al., 2019; Holzleitner et al., 2020; Patil et al., 2020; Widrich et al., 2021; Dinu et al., 2022）。随着时间的推移，它旨在发展自主生成自身计划的能力，高效安排任务和子任务，并仔细选择每个任务最合适的工具。我们的协议为该代理学习和扩展其基本能力集奠定了基础（Amaro et al., 2023），朝着更复杂的自我参考多步骤任务编排迈进。我们已经注意到研究正在向这种方法论转变（Yuan et al., 2024）。此外，在第 7 节中，我们仅考虑了一个顺序调度器。然而，我们的目标是最终评估一个非顺序任务执行模型，允许动态插入和非顺序任务执行。此外，我们还希望探索我们的工作与生成流网络（Bengio et al., 2021a;b; Lahlou et al., 2023）之间的相似性。最后，我们在附录第 8.2 节中讨论了局限性以及未来改进的进一步机会。

In this work, we introduced SymbolicAI, a framework that unifies generative models with an array of solvers, blending the strengths of symbolic and sub-symbolic AI paradigms within a cohesive NeSy framework. SymbolicAI equips researchers and practitioners with a comprehensive toolkit to develop contextualized and explainable NeSy AI systems capable of addressing diverse challenges effectively. We also introduce a quality measure and a benchmark for comparing and evaluating a wide range of computational tasks. SymbolicAI provides a basis for further research in advanced program synthesis, hierarchical computational graphs, the development of self-referential systems, and the integration of probabilistic methods with AI design for creating autonomous agents.

在本工作中，我们介绍了SymbolicAI，这是一个将生成模型与一系列求解器统一的框架，融合了符号和亚符号人工智能范式的优势，形成一个统一的NeSy框架。SymbolicAI为研究人员和实践者提供了一个全面的工具包，以开发能够有效应对多样化挑战的上下文化和可解释的NeSy人工智能系统。我们还引入了一种质量度量和基准，用于比较和评估广泛的计算任务。SymbolicAI为进一步研究高级程序合成、分层计算图、自指系统的发展以及将概率方法与人工智能设计相结合以创建自主代理提供了基础。

The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids (FFG- 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF, the NVIDIA Corporation and Atlas.

ELLIS 单元林茨、LIT AI 实验室、机器学习研究所得到了上奥地利联邦州的支持。我们感谢医疗认知计算中心（MC3）、INCONTROL-RL（FFG-881064）、PRIMAL（FFG-873979）、S3AI（FFG-872172）、DL for GranularFlow（FFG-871302）、EPILEPSIA（FFG-892171）、AIRI FG 9-N（FWF-36284，FWF-36235）、AI4GreenHeatingGrids（FFG-899943）、INTEGRATE（FFG-892418）、ELISE（H2020-ICT-2019-3 ID: 951847）、Stars4Waters（HORIZON-CL6-2021-CLIMATE-01-01）等项目的支持。我们感谢 Audi.JKU 深度学习中心、TGW LOGISTICS GROUP GMBH、Silicon Austria Labs (SAL)、FILL Gesellschaft mbH、Anyline GmbH、Google、ZF Friedrichshafen AG、Robert Bosch GmbH、UCB Biopharma SRL、Merck Healthcare KGaA、Verbund AG、GLS（滑铁卢大学）、软件能力中心哈根贝格 GmbH、Borealis AG、TÜV 奥地利、Frauscher Sensonic、TRUMPF、NVIDIA 公司和 Atlas 的支持。

We extend our appreciation to Andreas Windisch and Clemens Wasner of AI Austria for their unwavering support. Their valuable feedback, connections, and facilitation of introductions within their expansive network have been instrumental to the progress of ExtensityAI.

我们对 AI Austria 的 Andreas Windisch 和 Clemens Wasner 表示衷心感谢，感谢他们的坚定支持。他们宝贵的反馈、联系以及在其广泛网络中促进的介绍对 ExtensityAI 的进展起到了重要作用。

Our gratitude also goes to Sergei Pereverzyev, whose enlightened guidance and thoughtful ideas have been a beacon for our research endeavors. Our thanks are equally extended to Gary Marcus, whose stimulating discussions sparked numerous innovative ideas incorporated into our framework.

我们同样感谢 Sergei Pereverzyev，他的开明指导和深思熟虑的想法为我们的研究工作指明了方向。我们也要感谢 Gary Marcus，他的激励性讨论激发了许多创新想法，这些想法被纳入了我们的框架中。

We are equally grateful to Markus Hofmarcher, a friend and colleague whose informed counsel and stimulating discussions have significantly sharpened various facets of our study. Additionally, our thanks are due to Fabian Paischer and Kajetan Schweighofer, whose preliminary work and assistance have been of enormous benefit.

我们同样感谢马克斯·霍夫马赫（Markus Hofmarcher），一位朋友和同事，他的深思熟虑的建议和激发思考的讨论显著提升了我们研究的各个方面。此外，我们还要感谢法比安·派舍（Fabian Paischer）和卡耶坦·施韦戈费（Kajetan Schweighofer），他们的初步工作和帮助对我们有着巨大的益处。

We are also grateful to our friends John Chong Min Tan and Tim Scarfe, whose communities have been a hub for exhilarating discussions. Their online presence and engagement have enriched the AI research landscape and broadened our perspectives.

我们还要感谢我们的朋友约翰·钟敏·谭（John Chong Min Tan）和蒂姆·斯卡夫（Tim Scarfe），他们的社区成为了激动人心讨论的中心。他们的在线存在和参与丰富了人工智能研究的领域，拓宽了我们的视野。

Moreover, we wish to honor the memories of the cherished family members we lost in 2023. Their influence in our lives extended beyond personal bonds, and the principles they instilled in us continue to shape our journey. It is with great respect and affection that we acknowledge the indelible impact they have made, enabling us to persist in our scientific pursuits with determination and integrity.

此外，我们希望缅怀在2023年失去的亲人。他们在我们生活中的影响超越了个人纽带，他们在我们心中灌输的原则继续塑造着我们的旅程。我们怀着极大的尊重和深情，承认他们所留下的不可磨灭的影响，使我们能够以决心和诚信继续我们的科学追求。

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, 和 D. Amodei。语言模型是少量学习者。在 H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, 和 H. Lin (编)，《神经信息处理系统进展》，第 33 卷，页 1877-1901。Curran Associates, Inc.，2020。

F. Deniz, A. O. Nunez-Elizalde, A. G. Huth, and J. L. Gallant. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39):7722-7736, 2019. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.0675-19.2019.

F. Deniz, A. O. Nunez-Elizalde, A. G. Huth 和 J. L. Gallant. 在听与读的过程中，人类大脑皮层对语义信息的表征对刺激模态是不变的. 神经科学杂志, 39(39):7722-7736, 2019. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.0675-19.2019.

M.-C. Dinu, M. Hofmarcher, V. P. Patil, M. Dorfer, P. M. Blies, J. Brandstetter, J. A. Arjona-Medina, and S. Hochreiter. Xai and strategy extraction via reward redistribution. In A. Holzinger, R. Goebel, R. Fong, T. Moon, K.-R. Müller, and W. Samek (eds.), xxAI - Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers, pp. 177-205, Cham, 2022. Springer International Publishing. ISBN 978-3-031-04083-2. doi: 10.1007/978-3-031-04083-2_10.

M.-C. Dinu, M. Hofmarcher, V. P. Patil, M. Dorfer, P. M. Blies, J. Brandstetter, J. A. Arjona-Medina, 和 S. Hochreiter. 通过奖励重分配进行可解释人工智能和策略提取. 在 A. Holzinger, R. Goebel, R. Fong, T. Moon, K.-R. Müller, 和 W. Samek (编), xxAI - 超越可解释人工智能：国际研讨会, 与 ICML 2020 同期举行, 2020年7月18日, 奥地利维也纳, 修订和扩展论文, 第177-205页, Cham, 2022. 施普林格国际出版. ISBN 978-3-031-04083-2. doi: 10.1007/978-3-031-04083-2_10.

M.-C. Dinu, M. Holzleitner, M. Beck, H. D. Nguyen, A. Huber, H. Eghbal-zadeh, B. A. Moser, S. V. Pereverzyev, S. Hochreiter, and W. Zellinger. Addressing parameter choice issues in unsupervised domain adaptation by aggregation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.

M.-C. Dinu, M. Holzleitner, M. Beck, H. D. Nguyen, A. Huber, H. Eghbal-zadeh, B. A. Moser, S. V. Pereverzyev, S. Hochreiter, 和 W. Zellinger. 通过聚合解决无监督领域适应中的参数选择问题。在第十一届国际学习表征会议，ICLR 2023，卢旺达基加利，2023年5月1日至5日。OpenReview.net，2023年。

J. Hu, H. Small, H. Kean, A. Takahashi, L. Zekelman, D. Kleinman, E. Ryan, A. Nieto-Castañón, V. Ferreira, and E. Fedorenko. Precision fMRI reveals that the language-selective network supports both phrase-structure building and lexical access during language production. bioRxiv, 2022. doi: 10.1101/2021.09.10.459596.

J. Hu, H. Small, H. Kean, A. Takahashi, L. Zekelman, D. Kleinman, E. Ryan, A. Nieto-Castañón, V. Ferreira, 和 E. Fedorenko. 精确功能性磁共振成像揭示语言选择网络在语言生成过程中支持短语结构构建和词汇访问. bioRxiv, 2022. doi: 10.1101/2021.09.10.459596.

Z. Hu, X. Ma, Z. Liu, E. Hovy, and E. Xing. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2410-2420, Berlin, Germany, August 2016. Association for Computational Linguistics.

Z. Hu, X. Ma, Z. Liu, E. Hovy, 和 E. Xing. 利用逻辑规则驾驭深度神经网络. 在第54届计算语言学协会年会论文集（第1卷：长篇论文）中, 第2410-2420页, 德国柏林, 2016年8月. 计算语言学协会.

B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018

B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, 和 D. Amodei. 从人类偏好和演示中学习奖励，在Atari中。收录于 S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, 和 R. Garnett (编辑)，《神经信息处理系统进展》，第31卷。Curran Associates, Inc., 2018

B. Ibarz, V. Kurin, G. Papamakarios, K. Nikiforou, M. Abbana Bennani, R. Csordás, A. Dudzik, M. Bošnjak, A. Vitvit-skyi, Y. Rubanova, A. Deac, B. Bevilacqua, Y. Ganin, C. Blundell, and P. Velivčković. A generalist neural algorithmic learner. In LOG IN, 2022.

B. Ibarz, V. Kurin, G. Papamakarios, K. Nikiforou, M. Abbana Bennani, R. Csordás, A. Dudzik, M. Bošnjak, A. Vitvit-skyi, Y. Rubanova, A. Deac, B. Bevilacqua, Y. Ganin, C. Blundell, 和 P. Velivčković. 一种通用神经算法学习器。收录于 LOG IN, 2022.

G. Indiveri, B. Linares-Barranco, T. Hamilton, A. van Schaik, R. Etienne-Cummings, T. Delbruck, S. Liu, P. Dudek, P. HÃ€fliger, S. Renaud, J. Schemmel, G. Cauwenberghs, J. Arthur, K. Hynna, F. Folowosele, S. SAÃ GHI, T. Serrano-Gotarredona, J. Wijekoon, Y. Wang, and K. Boahen. Neuromorphic silicon neuron circuits. Frontiers in Neuroscience, 5, 2011. ISSN 1662-453X. doi: 10.3389/fnins.2011.00073.

G. Indiveri, B. Linares-Barranco, T. Hamilton, A. van Schaik, R. Etienne-Cummings, T. Delbruck, S. Liu, P. Dudek, P. HÃ€fliger, S. Renaud, J. Schemmel, G. Cauwenberghs, J. Arthur, K. Hynna, F. Folowosele, S. SAÃ GHI, T. Serrano-Gotarredona, J. Wijekoon, Y. Wang, 和 K. Boahen. 神经形态硅神经元电路. 神经科学前沿, 5, 2011. ISSN 1662-453X. doi: 10.3389/fnins.2011.00073.

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, 和 W. El Sayed. Mistral 7b. arXiv 预印本 arXiv:2310.06825, 2023.

N. Kassner, B. Krojer, and H. Schütze. Are Pretrained Language Models Symbolic Reasoners over Knowledge? In R. Fernández and T. Linzen (eds.), Proceedings of the 24th Conference on Computational Natural Language Learning, CoNLL 2020, Online, November 19-20, 2020, pp. 552-564. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.conll-1.45.

N. Kassner, B. Krojer, 和 H. Schütze. 预训练语言模型是否是知识的符号推理者? 在 R. Fernández 和 T. Linzen (编辑), 第24届计算自然语言学习会议论文集, CoNLL 2020, 在线, 2020年11月19-20日, 第552-564页. 计算语言学协会, 2020. doi: 10.18653/v1/2020.conll-1.45.

A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, and A. Mattick. Openassistant conversations - democratizing large language model alignment, 2023.

A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, S. ES, S. Suri, D. Glushkov, A. Dantuluri, A. Maguire, C. Schuhmann, H. Nguyen, 和 A. Mattick. Openassistant 对话 - 使大型语言模型对齐的民主化, 2023.

J. MacGlashan, M. K. Ho, R. Loftin, B. Peng, G. Wang, D. L. Roberts, M. E. Taylor, and M. L. Littman. Interactive Learning from Policy-Dependent Human Feedback. In Proceedings of the 34th International Conference on Machine Learning, pp. 2285-2294. PMLR, July 2017.

J. MacGlashan, M. K. Ho, R. Loftin, B. Peng, G. Wang, D. L. Roberts, M. E. Taylor, 和 M. L. Littman. 基于政策依赖的人类反馈的交互学习. 在第34届国际机器学习会议论文集中, 第2285-2294页. PMLR, 2017年7月.

Our work focuses on broad AI (Hochreiter, 2022) (see Figure 7) through the integration of symbolic and sub-symbolic AI methodologies. Broad AI extends beyond restricted focus on single-task performance of narrow AI. In broad AI, systems are engineered to handle a range of tasks with a high degree of autonomy, utilizing sensory input, accumulated experiences, and previously developed skills.

我们的工作集中在广泛的人工智能（Hochreiter, 2022）（见图7），通过整合符号和子符号人工智能方法。广泛的人工智能超越了对狭义人工智能单一任务表现的限制关注。在广泛的人工智能中，系统被设计为能够处理一系列任务，具有高度的自主性，利用感官输入、累积的经验和先前开发的技能。

Figure 7: Hierarchical model of "cognitive" abilities of AI systems (Chollet, 2019; Hochreiter, 2022). The figure of capabilities, such as knowledge transfer, interaction, adaptability, robustness, abstraction, advanced reasoning, and efficiency. Broad AI aims to mimic human cognitive adaptability and robustness through advanced methodologies like few-shot learning, self-supervised contrastive learning, and context-sensitive sensory processing. Notably, Broad AI applies principles such as conceptual short-term memory and modern Hopfield networks (Ramsauer et al., 2020) to better integrate context and memory, thus avoiding pitfalls like explaining away and short-cut learning. We acknowledge the potential of NeSy systems as a significant step towards AI systems capable of performing any cognitive task with human-like proficiency.

图7：人工智能系统“认知”能力的层次模型（Chollet, 2019；Hochreiter, 2022）。该图展示了知识转移、互动、适应性、鲁棒性、抽象、高级推理和效率等能力。广义人工智能旨在通过先进的方法，如少样本学习、自监督对比学习和上下文敏感的感知处理，模拟人类的认知适应性和鲁棒性。值得注意的是，广义人工智能应用了概念短期记忆和现代霍普菲尔德网络（Ramsauer et al., 2020）等原理，以更好地整合上下文和记忆，从而避免解释偏差和捷径学习等陷阱。我们承认NeSy系统的潜力是朝着能够以类人水平执行任何认知任务的人工智能系统迈出的重要一步。

NeSy methods form the basis for developing new cognitive architectures (Newell & Simon, 1956; Newell et al., 1957; Newell & Simon, 1972; Newell, 1990; Langley et al., 2009; Laird, 2022; Dawid & LeCun, 2023; Sumers et al., 2023; LeCun, 2022; Assran et al., 2023). This hybridization produces computational graphs capable of context-aware learning and reasoning, allowing AI to execute complex tasks with human-like flexibility.

NeSy方法为开发新的认知架构奠定了基础（Newell & Simon, 1956；Newell et al., 1957；Newell & Simon, 1972；Newell, 1990；Langley et al., 2009；Laird, 2022；Dawid & LeCun, 2023；Sumers et al., 2023；LeCun, 2022；Assran et al., 2023）。这种混合化产生了能够进行上下文感知学习和推理的计算图，使人工智能能够以类人灵活性执行复杂任务。

Borrowing nomenclature from Karpathy (2017); Dilhara et al. (2021), we refer to the next generation of software as Software 3.0, which consists of applications that are neither pre-determined at design time, nor learned through to neuromorphic circuits (Indiveri et al., 2011), however, purely established at inference time in the "thought" process of a NeSy system.

借用Karpathy（2017）；Dilhara等（2021）的术语，我们将下一代软件称为软件3.0，它由在设计时既未预先确定，也未通过神经形态电路（Indiveri et al., 2011）学习而完全在NeSy系统的“思维”过程中建立的应用程序组成。

With LLMs becoming more and more accessible, progress recently made possible by the vast open source contributions from Köpf et al. (2023); Touvron et al. (2023); Taori et al. (2023); Xu et al. (2023); Geng et al. (2023); Biderman et al. (2023), embedded accelerators for LLMs - or more generally NeSY engines - will be ubiquitous in future computation platforms, such as wearables, smartphones, tablets, consoles, or notebooks. Although current execution cycles are slow and error-prone, we expect to see further performance gains through improved operating system level optimizations, dedicated GPU-centric hardware refinement, and improved software interoperability. We believe that features for constructing and evaluating generative computational graphs. This includes but is not limited to compo-

随着大型语言模型（LLMs）变得越来越可及，最近的进展得益于Köpf等人（2023）；Touvron等人（2023）；Taori等人（2023）；Xu等人（2023）；Geng等人（2023）；Biderman等人（2023）的广泛开源贡献，LLMs的嵌入式加速器——或更一般的NeSY引擎——将在未来的计算平台中无处不在，例如可穿戴设备、智能手机、平板电脑、游戏机或笔记本电脑。尽管当前的执行周期缓慢且容易出错，但我们预计通过改进操作系统级别的优化、专用的以GPU为中心的硬件改进和改进的软件互操作性，将看到进一步的性能提升。我们相信构建和评估生成计算图的特性。这包括但不限于组合、可并行化和基于仿真的执行，具有多态操作和自指结构。在生成过程中，当前的编程语言通常在这些概念上存在不连贯或临时的解决方案。我们相信将这些概念的整体概率支持整合到现代软件和硬件中，将解锁新的编程范式，使其能够充分利用生成架构。我们希望社区将这些想法视为当代计算的重要组成部分。

Figure 9: The illustration demonstrates the integration of Neuro-Symbolic computation within the contemporary programming stack. Probabilistic programming paradigms are embedded natively alongside traditional languages and environments, facilitated by interfaces to various data sources, compute engines, and human interaction tools, streamlining their adoption in practical and research applications.

图9：该插图展示了神经符号计算在现代编程栈中的整合。概率编程范式与传统语言和环境原生嵌入，借助于与各种数据源、计算引擎和人机交互工具的接口，简化了它们在实际和研究应用中的采用。

self-instruction and self-referential sub-process evaluations, which enable the dynamic creation and evaluation of acquisition of complex concepts with only one or few examples at inference time. We believe this will enable the creation of autonomously self-evolving cognitive architectures (Langley et al., 2009; Dawid & LeCun, 2023; Sumers et al., 2023). We therefore see an inherent connection to generative design as an analogy for creating coherent and stable "thought" computational graphs, and believe this paves the path toward broad AI systems (see Section A) and is a requirement for developing artificial general intelligent agents.

自我指导和自我参考的子过程评估，使得在推理时仅用一个或少数几个示例即可动态创建和评估复杂概念的获取。我们相信这将使得自主自我演化的认知架构的创建成为可能（Langley et al., 2009; Dawid & LeCun, 2023; Sumers et al., 2023）。因此，我们看到生成设计与创建连贯且稳定的“思想”计算图之间存在内在联系，并相信这为广泛的人工智能系统铺平了道路（见A节），并且是开发人工通用智能代理的必要条件。

Finally, we also wish to express our concern about recent economic trends in the deep-tech industry, where we observe AI-related concentration of data and resources, coupled with a tendency towards closed-source practices. We strongly advocate for increased transparency and exchange of ideas to ensure a diverse and collective growth in our socioeconomic landscape. Therefore, we push towards a democratic and open-source initiative.

最后，我们还希望表达对深科技行业近期经济趋势的关注，在这里我们观察到与人工智能相关的数据和资源集中，以及向闭源实践的倾斜。我们强烈倡导增加透明度和思想交流，以确保我们社会经济环境中的多样性和集体增长。因此，我们推动民主和开源倡议。

where

σ

is the bandwidth parameter of the kernel and

∥ \cdot ∥

denotes the Euclidean norm. Utilizing

K

,we can now construct a measure of distance between distributions, by embedding them into the Reproducing Kernel Hilbert Space (RKHS) induced by

K

,using kernel mean embeddings. The resulting distance is called the Maximum Mean Discrepancy (MMD).

其中

σ

是核的带宽参数，

∥ \cdot ∥

表示欧几里得范数。利用

K

，我们现在可以通过将分布嵌入到由

K

诱导的再生核希尔伯特空间（RKHS）中，使用核均值嵌入来构建分布之间的距离度量。得到的距离称为最大均值差异（MMD）。

On the other hand, for multivariate Gaussian distributions, we can use the Fréchet distance as a measure of similarity, which is nothing but the associated Wasserstein-2 distance, for which an explicit formula is available in the Gaussian case. The resulting expression is as follows (Dowson & Landau, 1982):

另一方面，对于多元高斯分布，我们可以使用弗雷歇距离作为相似性的度量，这实际上就是相关的 Wasserstein-2 距离，对于高斯情况有明确的公式可用。得到的表达式如下（Dowson & Landau, 1982）：

Heuristically,at least for small

‖ \begin{matrix} μ_{1} - μ_{2} \end{matrix} ‖

,also the associated kernel evaluations

K (X_{1}, X_{2})

tend to be small (see also Hochreiter &Schmidhuber (1997)),which leads to a small MMD,if we ignore the terms associated to

K (X_{1}, X_{1})

K (X_{2}, X_{2})

(which cancel out due to same covariance structure).

从启发式上讲，至少对于小的

‖ \begin{matrix} μ_{1} - μ_{2} \end{matrix} ‖

，相关的核评估

K (X_{1}, X_{2})

也倾向于较小（参见 Hochreiter & Schmidhuber (1997)），这导致 MMD 较小，如果我们忽略与

K (X_{1}, X_{1})

、

K (X_{2}, X_{2})

相关的项（由于相同的协方差结构而相互抵消）。

Primitives In the SymbolicAI framework, at the core lies the concept of Primitives and the dynamic type creation of Symbol objects, which are central to inherit types of behaviors. Primitives are pre-defined operations that act on Symbol objects, encapsulating basic operations, such as arithmetic, logic, or casting operations, to name a few. These operations are crucial to the framework's versatility and form the foundation for more complex interactions within computational graphs. Essentially, they can be viewed as contextualized functions that accept a Symbol object, send it to the NeSy engine for evaluation, and return one or more new objects, primarily new symbols. One of the key features of operations is their polymorphism, which allows for them to be applied to various data types, such as strings, integers, floats, lists, and more, with different behaviors depending on the specific object instance. To execute operations, we utilize the Symbol object's value attribute containing the original data type. This will be then sent as a string representation to the engines to execute the needed operations. Consequently, all values are cast to a string representation. Remember, this was our implicit assumption (see Section 4). For custom objects, it is semantics.

在符号人工智能框架中，核心概念是原语及符号对象的动态类型创建，这些对象是行为继承类型的核心。原语是预定义的操作，作用于符号对象，封装了基本操作，如算术、逻辑或类型转换操作等。这些操作对于框架的多功能性至关重要，并构成了计算图中更复杂交互的基础。本质上，它们可以被视为上下文化的函数，接受一个符号对象，将其发送到NeSy引擎进行评估，并返回一个或多个新对象，主要是新符号。操作的一个关键特性是其多态性，这使得它们可以应用于各种数据类型，如字符串、整数、浮点数、列表等，具体行为取决于特定对象实例。为了执行操作，我们利用符号对象的值属性，该属性包含原始数据类型。然后，这将作为字符串表示发送给引擎以执行所需的操作。因此，所有值都被转换为字符串表示。请记住，这是我们的隐含假设（见第4节）。对于自定义对象，它是语义的。

Symbol Objects Creation and Dynamic Typing A Symbol object is a versatile entity that can encapsulate a variety of data types and behaviors. The creation of Symbol objects is facilitated through a metaclass, which enables the dynamic typing of these objects to inherit behaviors from a collection of primitives. This dynamic typing system is important for extending the functionality of Symbol objects beyond simple data containers; they contain specific behaviors appropriate for the operations they will perform. For instance, a Symb01 object may possess the behaviors of arithmetic computations, string manipulations, or even logical comparisons, depending on the defined primitives.

符号对象的创建和动态类型化符号对象是一个多功能实体，可以封装多种数据类型和行为。符号对象的创建通过一个 metaclass 来实现，该 metaclass 使这些对象的动态类型化能够从一组原语中继承行为。这种动态类型系统对于扩展符号对象的功能至关重要，超越简单的数据容器；它们包含适合于将要执行的操作的特定行为。例如，Symb01 对象可能具有算术计算、字符串操作或甚至逻辑比较的行为，具体取决于定义的原语。

Type Inheritance and Expression Creation Type inheritance in SymbolicAI is leveraged to create new expressions, which are specialized forms of Symb

\circ 1

objects designed to represent parts of a computational graph. Expressions extend the capabilities of Symbol objects by providing a structured way to define complex functionalities that can inherits the properties of Symbol objects while also being able to define its own unique behavior through a forward method, which is analogous to a computational graph node's evaluation function. Figure 10 gives an overview of an exemplary inheritance branch. Each Expression must feature a forward method, which must be overwritten to define its behavior. The inherited ...call ... method invokes the forward method, evaluating the expression and returning the result. This design pattern facilitates lazy evaluation of expressions, allowing for complex composition of expressions.

类型继承与表达式创建在SymbolicAI中，类型继承被用来创建新的表达式，这些表达式是专门设计的Symb

\circ 1

对象的特殊形式，用于表示计算图的部分。表达式通过提供一种结构化的方式来定义复杂功能，扩展了Symbol对象的能力，这些功能可以继承Symbol对象的属性，同时能够通过前向方法定义其独特的行为，这类似于计算图节点的评估函数。图10概述了一个示例继承分支。每个表达式必须具有一个前向方法，该方法必须被重写以定义其行为。继承的...call...方法调用前向方法，评估表达式并返回结果。这种设计模式促进了表达式的惰性评估，允许复杂的表达式组合。

Inherited from the Symbol class, the _sym_return_type and static_context properties establish the context in which the current Expression operates. The static_context impacts all operations of the current Expression subclass, while the _sym_return_type guarantees the acquisition of the desired return object type post-evaluation. Typically, this returns the current type but can be configured to return a different type. A more in-depth examination of both notions will be provided in the following section.

从Symbol类继承的_sym_return_type和static_context属性建立了当前表达式操作的上下文。static_context影响当前表达式子类的所有操作，而_sym_return_type则保证在评估后获得所需的返回对象类型。通常，这返回当前类型，但可以配置为返回不同的类型。对这两个概念的更深入的探讨将在下一节中提供。

Figure 10: Class diagram showing the inheritance and composition relationships among Symbol, Expression, and other inherited classes. Symbol serves as a base class for Expression where all the other types are derived from. Other types may contain or associate with zero or more Symbol types. For example, we illustrate how the Sequence derives from Expression and the multiplicity ’

0 \dots *

’ indicates that a Sequence can contain any number of Expression instances.

图10：类图显示了Symbol、Expression及其他继承类之间的继承和组合关系。Symbol作为Expression的基类，所有其他类型均从其派生。其他类型可以包含或关联零个或多个Symbol类型。例如，我们展示了Sequence如何从Expression派生，而多重性’

0 \dots *

’表示Sequence可以包含任意数量的Expression实例。

Utilizing Decorators for Operation Definition Decorators serve as a bridge between the declarative nature of symbolic operations and the imperative execution model of programming languages. By augmenting function definitions with decorators, the framework can dynamically assign operations to Symbol or Expression objects, which are then interpreted by the underlying NeSy engine or traditional solvers.

利用装饰器进行操作定义装饰器作为符号操作的声明性特征与编程语言的命令式执行模型之间的桥梁。通过用装饰器增强函数定义，框架可以动态地将操作分配给Symbol或Expression对象，这些对象随后由底层的NeSy引擎或传统求解器进行解释。

For example, the @core. logic decorator can be used to augment a Symbol object with the capability to perform logical and, or, or not operations contextually. Similarly, the @core . combine decorator allows the framework to define the semantics of combining or adding two symbolic values, regardless of their underlying data representations.

例如，@core.logic 装饰器可以用于增强 Symbol 对象，使其具备在上下文中执行逻辑与、或或非操作的能力。同样，@core.combine 装饰器允许框架定义组合或添加两个符号值的语义，而不考虑它们底层的数据表示。

Aspect-Oriented Programming The aspect-oriented programming paradigm offers a functional approach for extending or modifying the behavior of functions or methods without altering their code directly. This adheres to the principles of modularity and separation of concerns, as it allows for the isolation of specific functionalities while maintaining the original function's core purpose. By wrapping the original function, decorators provide an efficient and reusable way of adding or modifying behaviors. For instance, SymbolicAI integrates the zero- and few-shot learning with default fallback functionalities of pre-existing code.

面向切面编程面向切面编程范式提供了一种功能性的方法，用于扩展或修改函数或方法的行为，而无需直接更改其代码。这遵循模块化和关注点分离的原则，因为它允许在保持原始函数核心目的的同时，隔离特定功能。通过包装原始函数，装饰器提供了一种高效且可重用的方式来添加或修改行为。例如，SymbolicAI 将零样本和少样本学习与现有代码的默认回退功能集成在一起。

Adaptability: Through decorators we can easily modify or extend the behavior of operations without changing their core implementation. This flexibility facilitates the generation of adaptive workflows and reliable fallback mechanisms when experimental implementations do not fulfill required constraints.

适应性：通过装饰器，我们可以轻松修改或扩展操作的行为，而无需更改其核心实现。这种灵活性促进了自适应工作流的生成，以及在实验实现未满足所需约束时的可靠回退机制。

The Symbol class is an abstraction representing data and context. It holds not only the value itself, but metadata that guides its transformation and interpretation. Through inheritance and compositionality, the Symbol can be extended contain a reference to its parent and children, naturally forming a directed graph structure where the nodes are symbols and edges represent relationships between a symbol and its derivative computations.

Symbol 类是一个表示数据和上下文的抽象。它不仅包含值本身，还包含指导其转换和解释的元数据。通过继承和组合，Symbol 可以扩展以包含对其父节点和子节点的引用，自然形成一个有向图结构，其中节点是符号，边表示符号与其派生计算之间的关系。

The Linker class, is a metadata subclass, and tracks relationships and results, effectively annotating the graph with execution details. It keeps records of nodes' keys, allowing quick retrieval of related computational outcomes within the graph, and aids in tasks such as error tracing and debugging.

Linker 类是一个元数据子类，跟踪关系和结果，有效地为图形注释执行细节。它记录节点的键，允许快速检索图中相关的计算结果，并有助于错误追踪和调试等任务。

The computational graph's structure is further enriched by properties like nodes, edges, and graph itself, which collectively enable the comprehensive query of the computation's topology. These properties are used internally to enable graph visualizations, which are useful for debugging and analysis.

计算图的结构通过节点、边和图本身等属性进一步丰富，这些属性共同使得对计算拓扑的全面查询成为可能。这些属性在内部用于启用图形可视化，这对于调试和分析非常有用。

This command will install the latest version of SymbolicAI along with its core dependencies, enabling the integration of the framework into Python applications. If you intend to utilize the framework with local engines

^{5}

,or with engines powered by external APIs such as OpenAI's API, additional installation steps are required.

此命令将安装最新版本的 SymbolicAI 及其核心依赖项，使该框架能够集成到 Python 应用程序中。如果您打算将该框架与本地引擎

^{5}

或由外部 API（如 OpenAI 的 API）提供支持的引擎一起使用，则需要额外的安装步骤。

Before the first run, it is necessary to configure the required modules and optionally set necessary API keys to activate the respective engines. This can be done in multiple ways, but we recommend doing it through the configuration wizard by running this command in the terminal:

在第一次运行之前，需要配置所需的模块，并可选地设置必要的 API 密钥以激活相应的引擎。这可以通过多种方式完成，但我们建议通过在终端中运行以下命令来使用配置向导：

The SymbolicAI framework is designed to leverage multiple engines for a variety of operations. To fully utilize these capabilities, you may install additional dependencies or set up the optional API keys for specific engines like WolframAlpha, SerpApi, and others. In Figure 11 we conceptually outline the connection between the utilization of an LLM and its interact with other tools and solvers. Instructions and operations can be initiated by any user, pre-scripted knowledge base or learned meta agent.

SymbolicAI 框架旨在利用多个引擎进行各种操作。为了充分利用这些功能，您可以安装额外的依赖项或为特定引擎（如 WolframAlpha、SerpApi 等）设置可选的 API 密钥。在图 11 中，我们概念性地概述了 LLM 的使用与其与其他工具和求解器的交互之间的连接。指令和操作可以由任何用户、预先编写的知识库或学习的元代理发起。

In the context of LLMs, zero- and few-shot learning domains have emerged as essential techniques (Yao et al., 2023b; Shinn et al., 2023; Kim et al., 2023; Wei et al., 2022b; Lyu et al., 2023; Pitis et al., 2023; Madaan et al., 2023; Wang et al.,2022; Ye et al.,2023)

^{7}

to enable models to generalize from limited training data and adapt to new tasks without as it reduces the need for large labeled data sets and allows for faster deployment in new applications. In this section, we demonstrate how our Symbolic API incorporates Python decorators to define custom operations in the zero- and few-shot domains.

在大型语言模型（LLMs）的背景下，零样本和少样本学习领域已成为重要技术（Yao et al., 2023b; Shinn et al., 2023; Kim et al., 2023; Wei et al., 2022b; Lyu et al., 2023; Pitis et al., 2023; Madaan et al., 2023; Wang et al., 2022; Ye et al., 2023）

^{7}

，使模型能够从有限的训练数据中进行概括，并在不需要大量标记数据集的情况下适应新任务，从而加快在新应用中的部署。在本节中，我们展示了我们的符号 API 如何结合 Python 装饰器在零样本和少样本领域定义自定义操作。

In this example, the @symai . core. zero_shot decorator is used to define a custom operation that does not require any examples, as the prompt is expressive enough. The zero-shot decorator takes in two arguments: prompt and constraints. The prompt defines the conditioning for our desired operation behavior, while the constraints are used to validate the computed outcome, ensuring it meets our expectations. If the constraints are not fulfilled, the implementation would resort to the specified default implementation or the default value. If neither is provided, the The resulting value from the wrapped function must be of type int because of the specific implementation of the auto-casting realized through

\to

. If the cast fails,the Symbolic API raises a ValueError. If no return type is specified, the return type defaults to Any.

在这个示例中，@symai.core.zero_shot 装饰器用于定义一个不需要任何示例的自定义操作，因为提示足够表达。零样本装饰器接受两个参数：提示和约束。提示定义了我们期望的操作行为的条件，而约束用于验证计算结果，确保其符合我们的期望。如果未满足约束，实施将退回到指定的默认实现或默认值。如果两者都未提供，包装函数的结果值必须是 int 类型，因为通过

\to

实现的自动转换的具体实现。如果转换失败，符号 API 将引发 ValueError。如果未指定返回类型，返回类型默认为 Any。

The behavior of the prompt and constraints attributes is similar to the zero_shot decorator. The examples and limit arguments are new, with examples defining a list of instructions conditioning the NeSy engine, and limit specifying the maximum number of examples returned. The pre-processor and post-processor arguments accept lists of PreProcessor and Post Processor objects, respectively, which are utilized to process the input before being fed into the NeSy engine and the output before being returned to the user. The wrp_kwargs argument passes additional arguments to the wrapped method, streamlining them towards the NeSy engine, or other engines.

prompt 和 constraints 属性的行为与 zero_shot 装饰器相似。examples 和 limit 参数是新的，其中 examples 定义了条件化 NeSy 引擎的一系列指令，而 limit 指定返回的示例的最大数量。pre-processor 和 post-processor 参数分别接受 PreProcessor 和 Post Processor 对象的列表，这些对象用于在输入被送入 NeSy 引擎之前处理输入，以及在输出返回给用户之前处理输出。wrp_kwargs 参数将额外的参数传递给被包装的方法，从而简化它们向 NeSy 引擎或其他引擎的传递。

all the other Prompt classes. Consider the following example, where we define a Prompt for comparing two values through the NeSy engine. In it,when the

<=

operation on two Symbol objects will be resolved,the NeSy engine evaluates them in the context of the CompareValues prompt.

考虑以下示例，我们通过 NeSy 引擎定义一个用于比较两个值的提示。在这个示例中，当对两个 Symbol 对象的

<=

操作被解析时，NeSy 引擎在 CompareValues 提示的上下文中评估它们。

This evaluation returns True, as the fuzzy comparison operation conditions the engine to compare the two Symbol objects based on their semantic meaning. More generally, the semantics of Symbol operations may vary depending on the context hierarchy of the Expression class and the operations used. We used three main prompt designs: Context-based Prompts, Operational Prompts, and Templates. Prompts can be curated either through inheritance or composition. For instance, the static context can be defined by inheriting from the Expression class and overwriting the static_context property, while an Operation and Template prompt can be created by providing a PreProcessor to modify the input data.

此评估返回 True，因为模糊比较操作使引擎能够根据两个 Symbol 对象的语义意义进行比较。更一般地说，Symbol 操作的语义可能会根据 Expression 类的上下文层次结构和所使用的操作而有所不同。我们使用了三种主要的提示设计：基于上下文的提示、操作提示和模板提示。提示可以通过继承或组合进行策划。例如，静态上下文可以通过继承 Expression 类并重写 static_context 属性来定义，而操作和模板提示可以通过提供 PreProcessor 来修改输入数据。

Context-based Prompts are considered optional and can be defined in a static manner, either by sub-classing the Expression class and overwriting the static-context property, or at runtime by updating the dynamic_context property or passing a payload kwargs to a method. Below is an example of payload kwargs through the method signature:

基于上下文的提示被视为可选的，可以以静态方式定义，方法是通过子类化 Expression 类并重写 static-context 属性，或在运行时通过更新 dynamic_context 属性或将有效负载 kwargs 传递给方法。以下是通过方法签名传递有效负载 kwargs 的示例：

Operational Prompts define the behavior of an atomic operation and are therefore mandatory to express the nature of such an operation. For example, the + operation is used to add two Symbol objects together, and the + operation prompt explains its behavior. The examples kwargs provide another optional structure that conditions the NeSy engine with a set of instructions.

操作提示定义了原子操作的行为，因此对于表达此类操作的性质是必需的。例如，+ 操作用于将两个 Symbol 对象相加，而 + 操作提示解释了它的行为。示例 kwargs 提供了另一种可选结构，为 NeSy 引擎提供了一组指令。

Template Prompts are optional and encapsulate the resulting prediction to enforce a specific format. For example,to generate HTML tags,we can utilize a curated ... template. This template enforces the NeSy engine to begin the generation process already in the context of an HTML tags format and not produce irrelevant descriptions about its task.

模板提示是可选的，并封装了结果预测以强制特定格式。例如，为了生成 HTML 标签，我们可以利用策划的 ... 模板。该模板强制 NeSy 引擎在 HTML 标签格式的上下文中开始生成过程，而不是产生与其任务无关的描述。

The aforementioned example demonstrates the utilization of the causal-expression method, which allows us to extract information that can be resolved either manually or through external solvers, say WolframAlpha. Initially, when utilizing the GPT-3 backend, we anticipated a significant engineering effort to develop such a complex expression, as the GPT-3 backend frequently struggled with accurate information extraction and comparison resolution. However, we remained confident in the field's progress, specifically with fine-tuned models like RLHF ChatGPT. We were delighted to witness these challenges being further tackled through the latest GPT-4 model (OpenAI, 2023).

上述示例展示了因果表达法的应用，该方法允许我们提取可以通过手动或外部求解器（例如 WolframAlpha）解决的信息。最初，当使用 GPT-3 后端时，我们预计需要进行大量工程工作来开发如此复杂的表达式，因为 GPT-3 后端在准确的信息提取和比较解决方面经常遇到困难。然而，我们对该领域的进展保持信心，特别是针对像 RLHF ChatGPT 这样的微调模型。我们很高兴看到这些挑战通过最新的 GPT-4 模型（OpenAI, 2023）得到了进一步的解决。

Furthermore, it is worth highlighting that, given sufficient data, we could refine methods for information extraction or knowledge graph construction from natural language, enabling more intricate reasoning tasks, such as those previously mentioned. We also direct readers to recent publications on Text-to-Graph translations, especially the very influential CycleGT (Guo et al., 2020). This approach allows us to answer queries by simply traversing the graph and extracting the required information.

此外，值得强调的是，考虑到足够的数据，我们可以改进从自然语言中提取信息或构建知识图谱的方法，从而实现更复杂的推理任务，例如前面提到的那些任务。我们还建议读者参考最近关于文本到图翻译的出版物，特别是非常有影响力的 CycleGT（Guo 等，2020）。这种方法使我们能够通过简单地遍历图形并提取所需信息来回答查询。

Lastly, recent research (Kıcman et al., 2023; Ellis, 2023) has demonstrated that algorithms based on GPT-3.5 and GPT-4 establish new state-of-the-art accuracy on multiple causal benchmarks, while also exhibiting unique capabilities previously considered exclusive to humans, such as generating causal graphs and identifying background causal context from natural language. This points to the potential for LLMs to be used alongside existing causal methods as proxies for human domain knowledge, reducing human effort in setting up causal analyses and ultimately accelerating the widespread adoption of causal methods. Moreover, recent advances in LLMs have opened new frontiers for research, practice, and adoption of causal reasoning, transforming the way causal analysis is conducted and broadening the scope of applications for our framework.

最后，最近的研究（Kıcman 等，2023；Ellis，2023）表明，基于 GPT-3.5 和 GPT-4 的算法在多个因果基准上建立了新的最先进的准确性，同时还表现出以前被认为仅限于人类的独特能力，例如生成因果图和从自然语言中识别背景因果上下文。这表明大型语言模型（LLMs）有潜力与现有因果方法结合使用，作为人类领域知识的代理，减少人类在设置因果分析中的努力，并最终加速因果方法的广泛采用。此外，LLMs 的最新进展为因果推理的研究、实践和采用开辟了新的前沿，改变了因果分析的进行方式，并扩大了我们框架的应用范围。

One of the most prominent illustrations of this concept is exhibited by Word2Vec (Mikolov et al., 2013a). Word2Vec generates dense representations of words by training a shallow neural network to predict a word based on its neighboring words within a text corpus. These resulting vectors are extensively utilized in various natural language processing applications, including sentiment analysis, text classification, and clustering.

这一概念最显著的例子之一是 Word2Vec（Mikolov 等，2013a）。Word2Vec 通过训练一个浅层神经网络来预测一个词基于其在文本语料库中的邻近词，从而生成词的稠密表示。这些生成的向量广泛应用于各种自然语言处理应用中，包括情感分析、文本分类和聚类。

Drawing parallels with Word2Vec, our objective is to execute contextualized operations on different symbols. However, the key distinction lies in the fact that we operate within the natural language domain, as opposed to a vector space. Consequently, this grants us the capability to conduct arithmetic on words, sentences, paragraphs, and the like, while simultaneously validating the outcomes in a human-readable format.

与 Word2Vec 相比较，我们的目标是在不同符号上执行上下文化操作。然而，关键的区别在于我们在自然语言领域内操作，而不是在向量空间中。因此，这使我们能够对单词、句子、段落等进行算术运算，同时验证结果以人类可读的格式呈现。

In contrast to the Symbol object, the Expression represents a non-terminal symbol. It allows for further evaluation and extends the Symbol class by overwriting the ....call_____ method. It serves as the foundation for all other expressions and possesses additional capabilities, namely to fetch data from URLs, search the internet, or open files. These operations are intentionally separated from Symbol, as they do not utilize the value attribute of the Symbol class.

与 Symbol 对象相比，Expression 代表一个非终结符号。它允许进一步评估，并通过重写 ....call_____ 方法扩展 Symbol 类。它作为所有其他表达式的基础，并具有额外的功能，即从 URL 获取数据、搜索互联网或打开文件。这些操作与 Symbol 有意分开，因为它们不使用 Symbol 类的值属性。

Sequences offer a multitude of advantages in the realm of Expression objects, as they facilitate the creation of more sophisticated structural configurations. By embodying the Sequence expression, multiple expressions can be effectively evaluated at runtime, thus enhancing the flexibility, modularity, and adaptability of the framework.

序列在表达式对象领域提供了众多优势，因为它们促进了更复杂结构配置的创建。通过体现序列表达式，可以在运行时有效评估多个表达式，从而增强框架的灵活性、模块化和适应性。

Subsequently, this process can be recursively repeated on each summary node to construct a hierarchical clustering structure. As each node represents a summarized subset of the original information, the summary can function as an index. The resulting tree can be utilized to navigate and retrieve the original information, transforming the large data stream problem into a search problem. Alternatively, vector-based similarity searches can be employed to identify similar nodes. For searching within a vector space, dedicated libraries such as Annoy (Spotify, 2017), Faiss (Johnson et al., 2019), or Milvus (Wang et al., 2021a) can be used.

随后，这个过程可以在每个摘要节点上递归地重复，以构建层次聚类结构。由于每个节点代表原始信息的摘要子集，因此摘要可以作为索引。生成的树可以用于导航和检索原始信息，将大数据流问题转化为搜索问题。或者，可以使用基于向量的相似性搜索来识别相似节点。对于在向量空间内的搜索，可以使用专门的库，如 Annoy（Spotify，2017）、Faiss（Johnson 等，2019）或 Milvus（Wang 等，2021a）。

In summary, Stream expressions offer the advantage of processing large data streams in a more efficient and organized manner, while also enabling the integration with other expressions like Sequence and Cluster expressions. These combinations allow for a more effective approach to handling context limitations, promoting the extraction of meaningful information and improving the overall performance of the framework.

总之，流表达式提供了以更高效和有组织的方式处理大数据流的优势，同时还允许与其他表达式（如序列和聚类表达式）集成。这些组合使得处理上下文限制的方式更加有效，促进了有意义信息的提取，并提高了框架的整体性能。

Effective error handling and debugging are essential for ensuring the robustness and reliability of any software system, while explainability is essential for understanding the underlying behavior of the system, especially in the context of AI-driven applications. By developing a system that is both transparent and interpretable, we can gain valuable insights into the performance of the NeSy engines and identify potential areas for improvement.

有效的错误处理和调试对于确保任何软件系统的稳健性和可靠性至关重要，而可解释性对于理解系统的基本行为尤其重要，特别是在人工智能驱动的应用程序中。通过开发一个既透明又可解释的系统，我们可以获得对 NeSy 引擎性能的宝贵见解，并识别潜在的改进领域。

One of the fundamental aspects of the SymbolicAI API is being able to generate code. Consequently, errors may arise, and handling them contextually becomes vital. In pursuit of a self-evolving API, we introduce the Try expression, which includes built-in fallback statements and automatically retries execution after performing dedicated error analysis and correction. This expression analyzes both the input and the error, conditioning itself to resolve the error by modifying the original code

^{9}

. If the fallback expression succeeds,the result is returned; otherwise,the process is repeated for the specified number of retries. If the maximum number of retries is reached without resolving the issue, the error is raised again.

SymbolicAI API 的一个基本方面是能够生成代码。因此，可能会出现错误，处理这些错误的上下文变得至关重要。为了追求自我演化的 API，我们引入了 Try 表达式，该表达式包含内置的回退语句，并在进行专门的错误分析和修正后自动重试执行。该表达式分析输入和错误，调整自身以通过修改原始代码

^{9}

来解决错误。如果回退表达式成功，则返回结果；否则，过程将为指定的重试次数重复。如果达到最大重试次数仍未解决问题，则再次引发错误。

Consider the example of executing previously generated code that contains a syntax error. By employing the Execute expression, we can evaluate the generated code, which takes a symbol and proceeds with the execution. Despite the initial failure, the Try expression resolves the syntactic error, returning the corrected and evaluated code:

考虑执行包含语法错误的先前生成代码的示例。通过使用 Execute 表达式，我们可以评估生成的代码，该表达式接受一个符号并继续执行。尽管初始失败，Try 表达式解决了语法错误，返回修正和评估后的代码：

In a computational graph, the VERTEX score compares the distribution of the generated model answer at each node against a reference distribution by sampling multiple valid trajectories at each node for the reference distribution and accounting for randomness through some predefined random trajectories. For instance, one of the predefined random trajectories in our benchmark was the string of ASCII characters which are considered printable, namely 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$$&'( ) *+, - . / : ; <=>?@ [ ] ^_ { } \\ . Moreover, the VERTEX score is particularly well suited for the evaluation of multi-step

在计算图中，VERTEX 分数通过在每个节点对生成模型答案的分布与参考分布进行比较，来评估模型的表现。它通过在每个节点采样多个有效轨迹来获取参考分布，并通过一些预定义的随机轨迹来考虑随机性。例如，我们基准测试中的一个预定义随机轨迹是被认为可打印的 ASCII 字符串，即 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$$&'( ) *+, - . / : ; <=>?@ [ ] ^_ { } \\ 。此外，VERTEX 分数特别适合于多步骤的评估。

We defined a total of 15 tasks involving in-context associations between two Symbol instances. SymbolicAl's overloaded operators rely on predefined pseudo-grammars, as described in Section 4, that augment the operators with few-shot examples. For instance, the overloaded operator + utilized between two Symbol instances provides few-shot examples how to resolve additions with various data types:

我们定义了总共 15 个任务，涉及两个符号实例之间的上下文关联。SymbolicAl 的重载运算符依赖于预定义的伪语法，如第 4 节所述，这些伪语法通过少量示例增强了运算符。例如，在两个符号实例之间使用的重载运算符 + 提供了如何使用不同数据类型解决加法的少量示例：

We perform transformations between multiple modalities through language-based representations. Therefore, we need to evaluate the model's proficiency in tool utilization, classification and routing of requests to relevant modules. We define a multi-modal Expression to detect the category of a task based on its content and to forward the task to the appropriate tool. The expression creates interfaces to tools like WolframAlpha for mathematical expressions, Selenium for website content scraping, SerpApi for search queries, and APILayer for optical character recognition. Each of the five tests aims to evaluate the appropriate handling of a specific type of input by the multi-modal Expression type, such as processing a website URL for scraping, interpreting a search engine query, testing if two vectors are linearly independent, comparing large numbers, and extracting text from an image. The following example shows the MultiModalExpression implementation of the forward function that uses isinstance of operator on its own context to determine its current expression value and select the sub-routine that can evaluate the request.

我们通过基于语言的表示在多种模态之间进行转换。因此，我们需要评估模型在工具使用、请求分类和路由到相关模块方面的能力。我们定义了一个多模态表达式，以根据其内容检测任务的类别，并将任务转发给适当的工具。该表达式创建与工具的接口，例如 WolframAlpha 用于数学表达式，Selenium 用于网站内容抓取，SerpApi 用于搜索查询，以及 APILayer 用于光学字符识别。五个测试的每一个都旨在评估多模态表达式类型对特定类型输入的适当处理，例如处理用于抓取的网站 URL、解释搜索引擎查询、测试两个向量是否线性独立、比较大数字以及从图像中提取文本。以下示例展示了 MultiModalExpression 实现的 forward 函数，该函数在其自身上下文中使用 isinstance 运算符来确定当前表达式值并选择可以评估请求的子例程。

We designed three separate tests related to program synthesis, where each task assesses the ability of the models to generate and execute code based on natural language instructions or provided templates: 2) The second task tests the automatic code generation for API calls by fetching data from a specified URL and extracting specific information from the retrieved content. 3) The third task evaluates the ability to construct a custom Expression that processes a Symbol through a specific Function component from the SymbolicAI package. Each of the three tests follows a similar pattern, where the generated code is scored based on its similarity to valid references and normalized with random samples. Figure 13 shows possible samples from the third task category.

我们设计了三个与程序合成相关的独立测试，每个任务评估模型根据自然语言指令或提供的模板生成和执行代码的能力：2）第二个任务测试通过从指定 URL 获取数据并从检索到的内容中提取特定信息来进行 API 调用的自动代码生成。3）第三个任务评估构建一个自定义表达式的能力，该表达式通过 SymbolicAI 包中的特定函数组件处理一个符号。三个测试遵循类似的模式，生成的代码根据其与有效参考的相似性进行评分，并与随机样本进行归一化。图 13 显示了第三个任务类别的可能样本。

2）三个任务评估模型解决逻辑运算 AND、OR 和 XOR 的能力。例如，逻辑 AND 的测试结合了符号 Symbol（“喇叭只在星期天响”）和符号 Symbol（“我听到喇叭声”），并将答案与人类生成的参考“可能解决方案的喇叭”进行比较，解决方案空间中存在高度变异性。每个模型可能偏好不同的解决方案。

The following snippet shows how one can define a custom primitive class (CustomLogicPrimitive) for logic operators. The _____or_____ function gets overloaded and uses the built-in logic decorator from the core package to create a local function that evaluates two Symbol instances.

以下代码片段展示了如何为逻辑运算符定义一个自定义原始类（CustomLogicPrimitive）。_____or_____ 函数被重载，并使用核心包中的内置逻辑装饰器创建一个局部函数，该函数评估两个符号实例。

^{10}

,that preserves the original relationships. The following is a DSL snippet of the ’Who is Jay’s brother?' puzzle:

3）在另一个任务中，我们使用一个自定义表达式，该表达式定义了 DSL 语法和语义结构。我们使用这个表达式从自然语言陈述中提取高阶逻辑表达式，即谜题“谁是杰伊的兄弟？”

^{10}

，该表达式保留了原始关系。以下是“谁是杰伊的兄弟？”谜题的 DSL 代码片段：

4) 对于最终任务，我们再次使用谜题“杰伊的兄弟是谁？”来评估模型进行复杂转换的能力。我们使用 Z3 定理证明器（Moura & Bjørner, 2008）来解决“杰伊的兄弟是谁”这一谜题，条件是 Z3 求解器对爱因斯坦著名谜题“谁拥有鱼？”的解答。该任务涉及从自然语言到可执行代码的间接翻译，由 Z3 求解器完成；爱因斯坦谜题的解答作为一种自包含的“文档”，说明 Z3 求解器应如何使用。测试构建了一个模板，其中包括任务说明、谜题陈述以及对爱因斯坦谜题解答的引用。然后要求模型分析给定的问题和解答格式，并创建一个具有 Z3 语法的 Python 函数，以解决“杰伊的兄弟是谁？”的谜题。动态生成的代码在测试环境中通过 Python 的 exec 函数执行。我们检查对 Z3 求解器的访问，并运行生成的 solve_puzzle。

	Common Architecture
	Layers	Values
Backbone Output Layer Projection Layers Adaptation Layers	ResNet-18 (Adaptive Average Pooling Layer)	512
	Fully-connected Layer Batch Normalization 1D Layer ReLU Fully-connected Layer Batch Normalization 1D Layer ReLU Activation Dropout Layer Fully-connected Layer Batch Normalization 1D Layer ReLU Activation Dropout Layer Fully-connected Layer Batch Normalization 1D Layer ReLU Activation Dropout Layer	1024 1024 0.5 1024 0.5 1024 0.5
	Fully-connected Layer Batch Normalization 1D Layer ReLU Activation Dropout Layer Fully-connected Layer Batch Normalization 1D Layer ReLU Activation	512 0.5 512
Class Output Layer	CMD
Class Output Layer	Fully-connected Layer	5
Class Output Layer	MMD
Class Output Layer	Fully-connected Layer	5

	常见架构
	层	值
主干输出层投影层适应层	ResNet-18（自适应平均池化层）	512
	全连接层批量归一化 1D层 ReLU 全连接层批量归一化 1D层 ReLU 激活 Dropout层全连接层批量归一化 1D层 ReLU 激活 Dropout层全连接层批量归一化 1D层 ReLU 激活 Dropout层	1024 1024 0.5 1024 0.5 1024 0.5
	全连接层批量归一化 1D层 ReLU 激活 Dropout层全连接层批量归一化 1D层 ReLU 激活	512 0.5 512
类别输出层	CMD
类别输出层	全连接层	5
类别输出层	MMD
类别输出层	全连接层	5

	MDD Classifier
	Layers	Values
Backbone Output Layer	CMD/MMD-Method Adaptation Layer	512
Projection Layers	Fully-connected Layer Batch Normalization 1D Layer ReLU	512
Class Output Layer	Fully-connected Layer	5

Task	MMD
Task	SO	IWV	DEV	BPDA	TB
$CS \to Q$	$0.568 (\pm 0.007)$	$0.629 (\pm 0.023)$	$0.629 (\pm 0.023)$	$0.629 (\pm 0.023)$	$0.568 (\pm 0.007)$
$CS \to R$	$0.068 (\pm 0.009)$	$0.102 (\pm 0.020)$	$0.102 (\pm 0.020)$	$0.098 (\pm 0.006)$	$0.068 (\pm 0.009)$
$CS \to S$	$0.309 (\pm 0.010)$	$0.307 (\pm 0.001)$	$0.324 (\pm 0.021)$	$0.296 (\pm 0.012)$	$0.296 (\pm 0.012)$
$CS \to C$	$0.246 (\pm 0.016)$	$0.264 (\pm 0.013)$	$0.264 (\pm 0.013)$	$0.264 (\pm 0.013)$	$0.246 (\pm 0.016)$
$CS \to I$	$0.605 (\pm 0.012)$	$0.577 (\pm 0.004)$	$0.564 (\pm 0.001)$	$0.589 (\pm 0.021)$	$0.564 (\pm 0.001)$
$CS \to P$	$0.178 (\pm 0.006)$	$0.212 (\pm 0.012)$	$0.202 (\pm 0.011)$	$0.214 (\pm 0.007)$	$0.178 (\pm 0.006)$
Avg.	$0.329 (\pm 0.010)$	$0.349 (\pm 0.012)$	$0.348 (\pm 0.015)$	$0.348 (\pm 0.014)$	$0.320 (\pm 0.009)$

任务	MMD
任务	SO	IWV	DEV	BPDA	TB
$CS \to Q$	$0.568 (\pm 0.007)$	$0.629 (\pm 0.023)$	$0.629 (\pm 0.023)$	$0.629 (\pm 0.023)$	$0.568 (\pm 0.007)$
$CS \to R$	$0.068 (\pm 0.009)$	$0.102 (\pm 0.020)$	$0.102 (\pm 0.020)$	$0.098 (\pm 0.006)$	$0.068 (\pm 0.009)$
$CS \to S$	$0.309 (\pm 0.010)$	$0.307 (\pm 0.001)$	$0.324 (\pm 0.021)$	$0.296 (\pm 0.012)$	$0.296 (\pm 0.012)$
$CS \to C$	$0.246 (\pm 0.016)$	$0.264 (\pm 0.013)$	$0.264 (\pm 0.013)$	$0.264 (\pm 0.013)$	$0.246 (\pm 0.016)$
$CS \to I$	$0.605 (\pm 0.012)$	$0.577 (\pm 0.004)$	$0.564 (\pm 0.001)$	$0.589 (\pm 0.021)$	$0.564 (\pm 0.001)$
$CS \to P$	$0.178 (\pm 0.006)$	$0.212 (\pm 0.012)$	$0.202 (\pm 0.011)$	$0.214 (\pm 0.007)$	$0.178 (\pm 0.006)$
平均	$0.329 (\pm 0.010)$	$0.349 (\pm 0.012)$	$0.348 (\pm 0.015)$	$0.348 (\pm 0.014)$	$0.320 (\pm 0.009)$

	架构
	层
卷积块 1	一维卷积层批量归一化一维层 ReLU 激活最大池化一维层 Dropout
卷积块 2	一维卷积层批量归一化一维层 ReLU 激活最大池化一维层 Dropout
卷积块 3	一维卷积层批量归一化一维层 ReLU 激活最大池化一维层 Dropout
卷积块 3	自适应池化一维层
	方法详见表 7 和表 8。

Parameter Choice and Neuro-Symbolic Approaches for Deep Domain- Invariant Learning参数选择与深度领域不变学习的神经符号方法

Acknowledgments致谢

Abstract摘要

Marius-Constantin Dinu

PUBLICATIONS发表的论文

TEACHING教学

Contents目录

Chapter 1 Introduction第1章 引言

1.1 Domain Shift, Domain Adaptation and Domain- Invariant Learning1.1 领域转移、领域适应与领域不变学习

1.2 Model Selection and Parameter Choice Meth- ods for Unsupervised Domain Adaptation1.2 无监督域适应的模型选择和参数选择方法

1.3 In-Context Learning1.3 上下文学习

1.4 A Neuro-Symbolic Perspective on Large Lan- guage Models1.4 大型语言模型的神经符号视角

1.5 List of Publications1.5 发表的论文列表

Chapter 2 Selected Publications第二章 选定出版物

The balancing principle for parameter choice in distance-regularized domain adaptation距离正则化领域适应中的参数选择平衡原则

Abstract摘要

1 Introduction1 引言

2 Summary of results2 结果总结

3 Preliminaries3 初步知识

4 Balancing principle for domain adaptation4 领域适应的平衡原则

4.1 Balancing terms in a target error bound4.1 目标误差界限中的平衡项

4.2 Approximation of balancing value4.2 平衡值的近似

5 Empirical evaluations5 实证评估

5.1 Setup5.1 设置

5.2 Results5.2 结果

6 Conclusion and future work6 结论与未来工作

Broader impact更广泛的影响

Acknowledgments and Disclosure of Funding致谢与资金披露

References参考文献

Supplementary material for: The balancing principle for parameter choice in distance-regularized domain adaptation补充材料: 距离正则化领域适应中的参数选择平衡原则

1 Balancing value of target error bound1 目标误差界限的平衡值

2 Criterion for approximating the balancing value2 近似平衡值的标准

3 Generalization guarantee for balancing principle estimate3 平衡原则估计的泛化保证

4 MiniDomainNet dataset4 MiniDomainNet 数据集

5 Extended empirical evaluations5 扩展的实证评估

5.1 Details for training5.1 训练的详细信息

5.2 Details for computational resources and source code5.2 计算资源和源代码的详细信息

5.3 Details for model selection5.3 模型选择的详细信息

5.4 Results5.4 结果

6 Discussion of risks and limitations6 风险和局限性的讨论

Acknowledgments and Disclosure of Funding致谢与资金披露

References参考文献

ADDRESSING PARAMETER CHOICE ISSUES IN UNSU- PERVISED DOMAIN ADAPTATION BY AGGREGATION通过聚合解决无监督领域适应中的参数选择问题

ABSTRACT摘要

1 INTRODUCTION1 引言

2 RELATED WORK2 相关工作

3 AGGREGATION BY IMPORTANCE WEIGHTED LEAST SQUARES3 通过重要性加权最小二乘进行聚合

4 TARGET ERROR BOUND FOR ALGORITHM 14 算法 1 的目标误差界限

5 EMPIRICAL EVALUATIONS5 实证评估

5.1 EXPERIMENTAL SETUP5.1 实验设置

5.2 BASELINES5.2 基线

5.3 DATASETS5.3 数据集

5.4 RESULTS5.4 结果

6 CONCLUSION AND FUTURE WORK6 结论与未来工作

ACKNOWLEDGMENTS致谢

REFERENCES参考文献

A NOTATION AND PROOF OF MAIN RESULT主要结果的符号和证明

Summary of Notation符号总结

Problem问题

Proof of Lemma 1.引理 1 的证明。

B CONSTRUCTION OF FUNCTION SPACESB 函数空间的构造

C DATASETSC 数据集

D EXPERIMENTAL SETUPD 实验设置

D.1 COMPUTATIONAL RESOURCES AND IMPLEMENTATIONSD.1 计算资源和实现

D.2 ARCHITECTURES AND TRAINING SETUPD.2 架构与训练设置

D.3 MODEL SEQUENCED.3 模型序列

D.4 MATRIX INVERSIOND.4 矩阵求逆

D.5 Correlation Analysis: Target Accuracy of Individual Models vs. AGGREGATION WEIGHTD.5 相关性分析：单个模型的目标准确度与聚合权重

D.6 SENSITIVITY ANALYSIS: EFFECT OF ADDING INACCURATE MODELSD.6 敏感性分析：添加不准确模型的影响

D.7 Detailed EMPIRICAL RESULTSD.7 详细的实证结果

D.7.1 DETAILED SUMMARY RESULTSD.7.1 详细总结结果

D.7.2 DETAILED AMAZON REVIEWS RESULTSD.7.2 亚马逊评论结果的详细信息

D.7.3 Detailed MiniDomainNet ResultsD.7.3 MiniDomainNet 结果的详细信息

D.7.4 DETAILED TIME-SERIES RESULTSD.7.4 详细时间序列结果

SYMBOLICAI: A FRAMEWORK FOR LOGIC-BASED APPROACHES COMBINING GENERATIVE MODELS AND SOLVERSSYMBOLICAI：一个结合生成模型和求解器的基于逻辑的方法框架

ABSTRACT摘要

1 INTRODUCTION1 引言

2 RELATED WORK2 相关工作

3 PROBLEM DEFINITION3 问题定义

4 DESIGN PRINCIPLES4 设计原则

Parameter Choice and Neuro-Symbolic Approaches for Deep Domain- Invariant Learning
参数选择与深度领域不变学习的神经符号方法

Acknowledgments
致谢

Abstract
摘要

PUBLICATIONS
发表的论文

TEACHING
教学

Contents
目录

Chapter 1 Introduction
第1章引言

1.1 Domain Shift, Domain Adaptation and Domain- Invariant Learning
1.1 领域转移、领域适应与领域不变学习

1.2 Model Selection and Parameter Choice Meth- ods for Unsupervised Domain Adaptation
1.2 无监督域适应的模型选择和参数选择方法

1.3 In-Context Learning
1.3 上下文学习

1.4 A Neuro-Symbolic Perspective on Large Lan- guage Models
1.4 大型语言模型的神经符号视角

1.5 List of Publications
1.5 发表的论文列表

Chapter 2 Selected Publications
第二章选定出版物

The balancing principle for parameter choice in distance-regularized domain adaptation
距离正则化领域适应中的参数选择平衡原则

Abstract
摘要

1 Introduction
1 引言

2 Summary of results
2 结果总结

3 Preliminaries
3 初步知识

4 Balancing principle for domain adaptation
4 领域适应的平衡原则

4.1 Balancing terms in a target error bound
4.1 目标误差界限中的平衡项

4.2 Approximation of balancing value
4.2 平衡值的近似

5 Empirical evaluations
5 实证评估

5.1 Setup
5.1 设置

5.2 Results
5.2 结果

6 Conclusion and future work
6 结论与未来工作

Broader impact
更广泛的影响

Acknowledgments and Disclosure of Funding
致谢与资金披露

References
参考文献

Supplementary material for: The balancing principle for parameter choice in distance-regularized domain adaptation
补充材料: 距离正则化领域适应中的参数选择平衡原则

1 Balancing value of target error bound
1 目标误差界限的平衡值

2 Criterion for approximating the balancing value
2 近似平衡值的标准

3 Generalization guarantee for balancing principle estimate
3 平衡原则估计的泛化保证

4 MiniDomainNet dataset
4 MiniDomainNet 数据集

5 Extended empirical evaluations
5 扩展的实证评估

5.1 Details for training
5.1 训练的详细信息

5.2 Details for computational resources and source code
5.2 计算资源和源代码的详细信息

5.3 Details for model selection
5.3 模型选择的详细信息

5.4 Results
5.4 结果

6 Discussion of risks and limitations
6 风险和局限性的讨论

Acknowledgments and Disclosure of Funding
致谢与资金披露

References
参考文献

ADDRESSING PARAMETER CHOICE ISSUES IN UNSU- PERVISED DOMAIN ADAPTATION BY AGGREGATION
通过聚合解决无监督领域适应中的参数选择问题

ABSTRACT
摘要

1 INTRODUCTION
1 引言

2 RELATED WORK
2 相关工作

3 AGGREGATION BY IMPORTANCE WEIGHTED LEAST SQUARES
3 通过重要性加权最小二乘进行聚合

4 TARGET ERROR BOUND FOR ALGORITHM 1
4 算法 1 的目标误差界限

5 EMPIRICAL EVALUATIONS
5 实证评估

5.1 EXPERIMENTAL SETUP
5.1 实验设置

5.2 BASELINES
5.2 基线

5.3 DATASETS
5.3 数据集

5.4 RESULTS
5.4 结果

6 CONCLUSION AND FUTURE WORK
6 结论与未来工作

ACKNOWLEDGMENTS
致谢

REFERENCES
参考文献

A NOTATION AND PROOF OF MAIN RESULT
主要结果的符号和证明

Summary of Notation
符号总结

Problem
问题

Proof of Lemma 1.
引理 1 的证明。

B CONSTRUCTION OF FUNCTION SPACES
B 函数空间的构造

C DATASETS
C 数据集

D EXPERIMENTAL SETUP
D 实验设置

D.1 COMPUTATIONAL RESOURCES AND IMPLEMENTATIONS
D.1 计算资源和实现

D.2 ARCHITECTURES AND TRAINING SETUP
D.2 架构与训练设置

D.3 MODEL SEQUENCE
D.3 模型序列

D.4 MATRIX INVERSION
D.4 矩阵求逆

D.5 Correlation Analysis: Target Accuracy of Individual Models vs. AGGREGATION WEIGHT
D.5 相关性分析：单个模型的目标准确度与聚合权重

D.6 SENSITIVITY ANALYSIS: EFFECT OF ADDING INACCURATE MODELS
D.6 敏感性分析：添加不准确模型的影响

D.7 Detailed EMPIRICAL RESULTS
D.7 详细的实证结果

D.7.1 DETAILED SUMMARY RESULTS
D.7.1 详细总结结果

D.7.2 DETAILED AMAZON REVIEWS RESULTS
D.7.2 亚马逊评论结果的详细信息

D.7.3 Detailed MiniDomainNet Results
D.7.3 MiniDomainNet 结果的详细信息

D.7.4 DETAILED TIME-SERIES RESULTS
D.7.4 详细时间序列结果

SYMBOLICAI: A FRAMEWORK FOR LOGIC-BASED APPROACHES COMBINING GENERATIVE MODELS AND SOLVERS
SYMBOLICAI：一个结合生成模型和求解器的基于逻辑的方法框架

ABSTRACT
摘要

1 INTRODUCTION
1 引言

2 RELATED WORK
2 相关工作

3 PROBLEM DEFINITION
3 问题定义

4 DESIGN PRINCIPLES
4 设计原则

5 SYMBOLICAI FRAMEWORK
5 SYMBOLICAI 框架